Goals
In an effort to empower our MSFS Marketplace Partners and get their content released more quickly, we are making a change to our Marketplace content testing process. Effective March 24th, 2023, Marketplace Partners will own testing and signoff on the functionality of all content they submit for release. Microsoft will continue to own testing for compliance with Marketplace policies.
By distributing functional testing to the creators of content, the intent is to:
- Remove the current bottleneck with Microsoft doing all testing to determine release readiness.
- Allow our Marketplace Partners, who know their content best and have the most expertise with it, to make the call when functionality of their product meets the bar for release.
- Reduce communication round trips among Microsoft Test, Content Managers and Marketplace Partners in reporting issues and confirming whether they are release blockers or not.
To facilitate this change, we have set up separate segments in our Live environment for each partner to test their content prior to release. Every partner has their own dedicated segment and can only access the content associated with their seller ID/partner account, both released and unreleased. Partners will be asked to specify gamertags of internal team members who need to be granted access to their testing segment.
Policies
The following policies are being implemented along with this process:
- It is not an option for partners to keep functional testing with Microsoft. Partners are required to take ownership of their content’s functional testing.
- Partners are required to use Xbox Series X/S hardware or Xbox Cloud Gaming to test their own Xbox content. We will gather information from partners on whether they are using Xbox Series X/S hardware or Xbox Cloud Gaming for functional testing. Partners are welcome but not required to use a combination of these Xbox testing mechanisms.
- Microsoft will not attempt to reproduce functional issues reported by partners.
- Content will be handed off to partners for functional testing by Monday of each week.
- In order for content to be included in a given week’s Thursday release, partners must provide functional signoff by Wednesday at 9 AM PT in the same week. Microsoft’s compliance signoff must also be completed for release to proceed.
- Detailed functional test cases for each content type are outlined later in this document. Partners are expected to complete all test cases for every platform on which they intend to release their content, and verify that results are sufficient for release prior to providing functional signoff.
- Microsoft will monitor for major issues found with content after release, including those that require emergency updates and/or delisting the products in question. Any partner who has such issues surface on a recurring basis will be subject to a cooling-off period.
- We seek to provide a global product and will continue to localize Marketplace and Hangar descriptions. However, we are no longer gating initial product release on localization being complete.
Process Workflow
Functional Test Cases to be Covered by MSFS Marketplace Partners
Aircraft/Livery – In Sim
| Test Case ID | Test Case Description |
| Aircraft Test 1 | Verify content price matches expected value. |
| Aircraft Test 2 | Verify content can be downloaded as a full package. |
| Aircraft Test 3 | If an update, verify content can be updated successfully from previous package. |
| Aircraft Test 4 | After downloading, verify in Content Manager that content was successfully downloaded. |
| Aircraft Test 5 | Verify content loads into sim in correct initial state. |
| Aircraft Test 6 | Verify there are no major performance issues. |
| Aircraft Test 7 | Verify there are no major graphical issues (textures/geometry). |
| Aircraft Test 8 | Verify there are no major LOD issues. |
| Aircraft Test 9 | Verify engine can be turned off and on without issue. |
| Aircraft Test 10 | Verify yoke can be moved or hidden if interactable controls are covered. |
| Aircraft Test 11 | Verify there is at least one pilot present. |
| Aircraft Test 12 | Verify there are no obvious issues with the flight model (lift, drag, handling/performance. |
| Aircraft Test 13 | Verify flight surfaces (flaps, trims, rudder, ailerons + elevators) can be adjusted and animations appear accurate. |
| Aircraft Test 14 | Verify audio/SFX is present and accurate in all states (idle, takeoff, cruising, engine off). |
| Aircraft Test 15 | Verify all expected controls are interactable without issue. |
| Aircraft Test 16 | Verify there are no issues with labels of interactable controls. |
| Aircraft Test 17 | Verify there are no issues with taxi functionality (moving and braking). |
| Aircraft Test 18 | Verify any available flight displays (e.g. G1000) are interactable without issue. |
| Aircraft Test 19 | Verify cockpit and HUD elements function correctly (glass HUD, glass cockpit). |
| Aircraft Test 20 | Verify HUD in external view is functional and information is accurate (engine, flaps degrees). |
| Aircraft Test 21 | Verify there are no major formatting issues with loading screen tips. |
| Aircraft Test 22 | Verify all external or cabin lights display without issue in any camera view. |
| Aircraft Test 23 | Verify there are no unexpected items or inappropriate images (guns/weapons, language). |
| Aircraft Test 24 | Verify weapons do not appear on aircraft when adjusting weight settings in Weight and Balances menu. |
| Aircraft Test 25 | Verify flight can be restarted. |
Activity – In Sim
| Test Case ID | Test Case Description |
| Activity Test 1 | Verify content price matches expected value. |
| Activity Test 2 | Verify content can be downloaded as a full package. |
| Activity Test 3 | If an update, verify content can be updated successfully from previous package. |
| Activity Test 4 | After downloading, verify in Content Manager that content was successfully downloaded. |
| Activity Test 5 | Verify content loads into sim in correct initial state. |
| Activity Test 6 | Verify there are no major performance issues. |
| Activity Test 7 | Verify there are no major graphical issues (textures/geometry). |
| Activity Test 8 | Verify there are no major LOD issues. |
| Activity Test 9 | Verify there are no major issues with UI or HUD. |
| Activity Test 10 | Verify additional content included functions as expected. |
Scenery – In Sim
| Test Case ID | Test Case Description |
| Scenery Test 1 | Verify content price matches expected value. |
| Scenery Test 2 | Verify content can be downloaded as a full package. |
| Scenery Test 3 | If an update, verify content can be updated successfully from previous package. |
| Scenery Test 4 | After downloading, verify in Content Manager that content was successfully downloaded. |
| Scenery Test 5 | Verify content loads into sim in correct initial state. |
| Scenery Test 6 | Verify there are no major performance issues. |
| Scenery Test 7 | Verify there are no major graphical issues (textures/geometry). |
| Scenery Test 8 | Verify there are no major LOD issues. |
| Scenery Test 9 | Verify content matches description of what is provided. |
| Scenery Test 10 | Verify scenery has POI markers. |
| Scenery Test 11 | Verify scenery appears to fit in with surrounding area. |
| Scenery Test 12 | Confirm no weapons are present. |
Airport – In Sim
| Test Case ID | Test Case Description |
| Airport Test 1 | Verify content price matches expected value. |
| Airport Test 2 | Verify content can be downloaded as a full package. |
| Airport Test 3 | If an update, verify content can be updated successfully from previous package. |
| Airport Test 4 | After downloading, verify in Content Manager that content was successfully downloaded. |
| Airport Test 5 | Verify content loads into sim in correct initial state. |
| Airport Test 6 | Verify there are no major performance issues. |
| Airport Test 7 | Verify there are no major graphical issues (textures/geometry). |
| Airport Test 8 | Verify there are no major LOD issues. |
| Airport Test 9 | Verify content matches description of what is provided. |
| Airport Test 10 | Verify airport has POI marker. |
| Airport Test 11 | Verify airport is available on World Map after downloading and name is accurate. |
| Airport Test 12 | Verify there are no major issues departing from different runways and parking spaces. |
| Airport Test 13 | Verify there are no major issues arriving at different runways. |
| Airport Test 14 | Verify airport appears correct at all times of day and with different weather. |
| Airport Test 15 | Verify there are no unexpected items or inappropriate images (guns/weapons, language). |