RFC: Upcoming Firmware Rework (Devs: This is a must-read!)

[Ancyker]

If you are someone who contributes code to the repository, reading this post in its entirety is highly recommended. If you aren't a dev, the tldr is the firmware is about to change in a big way and there are likely to be bugs, so continuing to use master after the firmware rework is pushed is discouraged unless you want to help with testing.

As some of you likely know, I've been working on a complete overhaul of the current firmware for over a year now. I had several goals when starting this endeavor, including, but not limited to:

• Move all functions and variables into namespaces rather than having everything in the global scope.
• Condense redundant code where possible.
• Make it easier to maintain multiple hardware versions, namely, reduce the heavy use of inline preprocessor directives for different displays/etc (OLED/LCD/serial).
• Add support for translating the UI.
• Reduce the (ab)use of global variables.
• Switch to a database format that's friendlier to embedded devices.
• Support mixing features from various hardware versions (i.e. the OLED display on a HW5, serial on any hardware version, etc).
• Document the code as much as possible (via Doxygen).
• Use classes where it makes sense to (but also don't make things classes just for the sake of it).
• Make the compiled binary smaller (hopefully).
• Begin the process of moving off of the Arduino IDE (to PlatformIO).

I'm happy to report that, at this time, all of these goals appear to have been attained. At this point, I'm just doing some touch ups, making sure things are documented properly, etc. I need to merge the last few weeks/months of changes as well, but after that, the rework will be ready to push.

The rework will have bugs. I can't test every single feature -- I don't have most of the adapters or cartridges to test them with even if I did. All I can test is that they compile. Beyond that, it will be up to the community to test them and report back with any bugs they find. Hopefully, most of the bugs that appear will be cosmetic, UI-related, bugs (i.e. missing titles/headers, being prompted to press a button to continue multiple times, not being asked at all, etc) rather than "it just doesn't work" type of bugs. Hopefully.

For you developers, the bad news is that literally nothing is the same in terms of function/variable names. The good news with that is the new names are still pretty intuitive (i.e. println_Msg(...) is now OSCR::UI::printLine(...)). There are also new functions/methods that combine very common things into a single function (i.e. OSCR::UI::printLineSync(...) is both printLine and update). Also, as you may have guessed, everything to do with the UI is scoped under the OSCR::UI namespace.

Additionally, it is now easier than ever to make menus, progress bars, etc. Here's an example of a real menu (N64):

  // N64 start menu
  void menu()
  {
    openCRDB();

    if (!refreshCart())
    {
      closeCRDB();
      return;
    }

    do
    {
      switch (static_cast<MenuOption>(OSCR::UI::menu(FS(OSCR::Strings::Cores::N64), menuOptions, sizeofarray(menuOptions))))
      {
      case MenuOption::Cart:
        cartMenu();
        continue;

      case MenuOption::Controller:
        controllerMenu();
        continue;

# if HAS_FLASH
      case MenuOption::FlashRepro:
        flashRepro();
        if (!refreshCart()) continue;
        cartMenu();
        continue;
# endif

      case MenuOption::RefreshCart:
        if (!refreshCart()) continue;
        break;

      case MenuOption::FlashGameshark:
        if (!flashGameshark()) continue;
        if (!refreshCart()) continue;
        break;

      case MenuOption::FlashXplorer64:
        flashXplorer();
        continue;

      case MenuOption::Back:
        closeCRDB();
        return;
      }

      OSCR::UI::waitButton();
    }
    while (true);
  }

A quick explanation of the above (in the order you read the code in):

• openCRDB() opens the cartridge database.
• refreshCart() is called to check if the cartridge is present and valid. While not all cores have it implemented yet, there is a retry function available. If this function returns false, you will be returned to the previous menu (after closing the database file).
• We begin an infinite loop to handle menu navigation.
• We call OSCR::UI::menu(...), in this case we are using a flash string as the menu header, and a flash array of flash strings as the menu options. This new menu function automatically handles everything else -- including if multiple pages are needed to display every option. The only thing you need to worry about when using it is handling the result. Regardless of whether the output is the LCD, OLED, or serial, it works the same way here.
• The menu function returns a value (uint8_t) which is the index of the selected option. We cast it to the MenuOption type, which is just a simple enum. This isn't needed here, but it makes what's happening a bit more clear and allows people to reorder the menu without touching this part of the code. You'd just need to change the array and enum to change the order of the menu options.
• We then have a bunch of cases for the various options. Because we are in an infinite loop inside of a function, we can use continue to run the loop again, or return to exit the function and go back a menu (to the main menu, in this case). Because we are inside of a switch statement, break exits that and the loop continues.
• OSCR::UI::waitButton() will print the text "Press a button to continue" and wait until the user does. This message will always be printed on the last line of the output if it has a fixed number of lines (LCD, etc), or after an empty new line if using an output that doesn't have fixed lines (serial).
• After that function returns, the loop begins again.

As you likely can see, this function is not only fairly simple to work with, but also provides a much better user experience than the previous method did as it does not require resetting to go back a menu.

This is just one of many improvements. As they are fairly intuitive and the cores provide many real-world examples, I won't go into detail for all of them, but I will show a few more at once and let you decide how easy it is to understand (written specifically as an example because most functions that use this stuff are very long):

void readROM()
{
  OSCR::UI::printHeader(FS(OSCR::Strings::Cores::GameBoy)); // GB just used as an example

  OSCR::Storage::Shared::createFile(FS(OSCR::Strings::FileType::GameBoy), FS(OSCR::Strings::Directory::ROM), fileName);

  cartOn();

  OSCR::UI::ProgressBar::init(romSize, 1);

  OSCR::UI::printSync(FS(OSCR::Strings::Status::Reading)); // "Reading... "

  for (uint16_t position = 0; position < romSize; position += 512)
  {
    for (uint16_t i = 0; i < 512; i++)
    {
      OSCR::Storage::Shared::buffer[i] = readByte(position + i);
    }

    OSCR::Storage::Shared::dump();
    OSCR::UI::ProgressBar::advance(512);
  }

  cartOff();

  OSCR::Storage::Shared::close();

  OSCR::UI::printLine(FS(OSCR::Strings::DONE)); // "DONE" (now "Reading... DONE" with a new line)

  OSCR::UI::ProgressBar::finish(); // Fills the progress bar to 100% (if needed), then goes to the next line.
}

Additionally, I've created a new (optional) bootloader for the ATmega2560. There are a few variants of it that will be included (hex files, source files, and a build script) with the rework. There are a few situations where you might want to use one of these modified bootloaders, the biggest of which is that all of the variants are smaller than the usual bootloader. This frees up program memory which allows you to enable more system cores at once.

A more technical reason to use one of the new bootloaders is if you have VSELECT or get the upcoming hardware revision with an integrated ATmega2560. With the former, when flashing an update, it will use 5V mode for more reliable flashing. When not flashing an update, it will set it to 3.3V immediately. For the latter, it adds support the the new RGB LED and CBUS; it is borderline required when using an on-board mega rather than a module and is actually required when using the new CBUS feature.

With all the good stuff out of the way, let's get to the down sides of it. Other than the already mentioned bugs, the main down sides to the rework are:

• The biggest change that will affect the most people is the gcc version that the Arduino IDE ships with needs updated for the best experience. It's technically optional, but highly recommended.
  • The minimum C++ standard that will be supported is gnu++1z. While that version is supported by the gcc included with the IDE, it is not the standard it uses by default, so it will need changed manually. Also, when using gnu++1z, some features cannot be enabled because they make use of things that are not supported by that standard.
  • The target version is c++20. Using this version will result in the smallest binary, but requires updating the Arduino IDE's gcc.
• Most of the reworked code was moved to a subfolder to keep it organized. The Arduino IDE does not show files in subfolders, so you can't edit these files within the Arduino IDE. The cores are still in the root directory as .ino files, same as they are now. So, this doesn't affect anyone who is only working on system cores. It only matters for the OSCR internals (UI, RTC, etc).
• I did not implement the N64 controller test yet. It requires some thinking on what the best way to do it would be, as it needs to work on multiple outputs. This is especially important as I wanted to make a controller adapter so you could test other controllers, so it needs to be done in a way that it can be reused for those.

I've uploaded the generated documentation to OSCR.tools. Not all functions/features are fully documented, some documentation is incorrect, and some things are still being changed around/worked on. However, reviewing it should still give you a basic idea of what to expect.

As for a timeline, I'd currently estimate that it will be ready before the end of the month -- possibly even sometime during this coming week. Feel free to ask any questions, make any requests, etc, related to the rework in this thread or in Discord.

[sanni]

I created an organization: https://github.com/oscartreader

This should give us more options to organize stuff, especially if we might venture to a different microcontroller in the future. 🚀
I will need help with the setup though, it's empty right now. 😅