2.1 Task Definition

Participants in the CDX track of SDX’23 were asked to submit systems that can extract the dialogue $sDX(n)∈ℝ2$M9 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[{{\bf{s}}_{{\rm{DX}}}}(n) \in \mathbb{R}^{2}\] \end{document} , sound effects $sFX(n)∈ℝ2$M10 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[{{\bf{s}}_{{\rm{FX}}}}(n) \in \mathbb{R}^{2}\] \end{document} , and music $sMX(n)∈ℝ2$M11 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[{{\bf{s}}_{{\rm{MX}}}}(n) \in \mathbb{R}^{2}\] \end{document} from the stereo cinematic audio

where n denotes the time index and all stereo signals are sampled at 44.1 kHz. We used the following definition for each class:³

• Dialogue refers to all spoken content in a movie including conversations between characters, monologues, and any other spoken elements.
• Sound effects are sounds that are used to support or complement the action on screen. They can be split into object sounds (e.g., footsteps) and ambient sounds (e.g., wind or rain).
• Music refers to the soundtrack that accompanies the visuals and which is often used to provide an emotional context. It might be a single instrument (e.g., a violin in a dramatic moment) or a full orchestra or band.

We verified these definitions with mixing engineers from Sony Pictures.

A unique aspect of this challenge was the requirement for participants to submit their pre-trained models along with the corresponding inference code, as the test dataset was kept hidden. This stands in contrast to many other challenges where participants have access to unlabeled test data and are required to submit processed files or labels.

2.2 Leaderboards

Submissions were categorized under two leaderboards:

• Leaderboard A was designated for models exclusively trained on the train and validation splits ‘tr’ and ‘cv’ of the Divide and Remaster (DnR) dataset (Petermann et al., 2022), while
• Leaderboard B was for models trained on any data.

The rationale behind this dual-leaderboard approach is threefold. Firstly, it allows individuals who may not have access to extensive datasets to participate in the competition. Secondly, it provides a platform to explore data augmentation strategies, such as mono-to-stereo conversion, which is particularly relevant as the DnR dataset is monaural, while the hidden test set used for evaluation is in stereo format. Thus, the two leaderboards not only foster inclusivity but also encourage innovative approaches to data augmentation. Thirdly, the two leaderboards allow disentangling data improvements from algorithm improvements, as Leaderboard B performance could come from extra data or better augmentation strategies relying on additional data (e.g., room impulse responses), while Leaderboard A improvements must come from augmentations without additional data and from algorithms only. However, Leaderboard B is required to determine the true state of the art.

2.3 Ranking Metric

For the evaluation of the systems, we used the global signal-to-distortion ratio (SDR) which is defined for one movie clip as

with $SDRj=10log10Σn‖sj(n)‖2Σn‖sj(n)−s^j(n)‖2$M12 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[{\rm{SD}}{{\rm{R}}_j} = 10{\log _{10}}{\textstyle{{{\Sigma _n}{{\bf{s}}_j}(n){^2}} \over {{\Sigma _n}{{\bf{s}}_j}(n) - {{{\bf{\hat s}}}_j}(n){^2}}}}\] \end{document} where $sj(n)∈ℝ2$M13 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[{{\bf{s}}_j}(n) \in \mathbb{R}^{2}\] \end{document} and $s^j(n)∈ℝ2$M14 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[{{\bf{\hat s}}_j}(n) \in \mathbb{R}^{2}\] \end{document} denote the stereo target and estimate for source j ∈ {DX,FX,MX}. The definition in Equation (2) is also called utterance-level SDR (cf., for example, Luo and Yu, 2023) and equivalent to the SDR of multi-channel BSS Eval v3 (Vincent et al., 2007). Finally, the global SDR of (2) is averaged over all clips in the hidden test dataset and the three sources DX, FX, and MX to obtain the final score. We chose this metric to rank submissions over scale-invariant metrics like SI-SDR (Le Roux et al., 2019), because systems with good SDR performance have the advantage that they can easily be blended with other models (Uhlich et al., 2017) and also allow one to compute the residual $s^¬j(n)=x(n)−s^j(n)$M15 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[{{\bf{\hat s}}_{\neg j}}(n) = {\bf{x}}(n) - {{\bf{\hat s}}_j}(n)\] \end{document} without having to recover the correct scale.

Besides the chosen global SDR (2), there are also other metrics that were proposed in the literature for the comparison of source separation models. As part of MDX’21, a thorough comparison of different metrics was performed by Mitsufuji et al. (2022) to show that Equation (2) highly correlates with many other metrics, in particular those that were used in previous iterations of the SiSEC competition in 2015, 2016, and 2018. We refer the interested reader to Mitsufuji et al. (2022) for more details.

2.4 Timeline, Challenge Phases and Prizes

The challenge took place in two phases. Phase I started on January 23rd, 2023. Phase II commenced on March 6th, 2023, as planned. However, due to the submission system experiencing difficulties in handling the surge in the number of submissions towards the end of the challenge, the end date of Phase II was extended by one week, to May 8, 2023, to ensure a fair competition for all teams.

CDXDB23 was partitioned into three sets of approximately equal size, containing three, three, and four movies respectively. During Phase 1 of the competition, participants were able to assess the performance of their submissions using one-third of the movies from the hidden test set. In Phase 2, this was expanded to include two-thirds of the movies from the hidden test set. Upon the conclusion of Phase 2, participants were required to select three submissions for evaluation on the full hidden test set, the results of which were then displayed on the final leaderboards. This selection process was implemented to mitigate the potential impact of overfitting. In cases where participants did not explicitly select three submissions, the top three submissions from the Phase 2 leaderboard were automatically chosen for final evaluation.

For Leaderboard A, which was for models trained exclusively on the Divide and Remaster (DnR) dataset, a total of 5,000 USD was distributed among the top three submissions. The first-place winner received 2,500 USD, the second-place winner was awarded 1,500 USD, and the third-place winner received 1,000 USD. To be eligible for these prizes, participants were required to open-source both their training and inference code as well as the pretrained model. Similarly, for Leaderboard B, which was for models trained on any data, the same prize distribution was applied. For this leaderboard, participants were required to open-source their inference code as well as the pretrained model. Compliance with these open-source requirements was ensured by the organizers through a due diligence check. In the course of this evaluation, a thorough review of the source code was conducted to verify that participants in Leaderboard A exclusively trained their models using only DnR.

3.1 Divide and Remaster (DnR) – Training Dataset

Introduced by Petermann et al. (2022), the Divide and Remaster (DnR) dataset serves as a tool for developing and evaluating mono audio signal separation algorithms applied to podcasts, television, and movies. It includes artificial mixtures sourced from LibriSpeech (Panayotov et al., 2015), Free Music Archive (FMA) (Defferrard et al., 2016), and Freesound Dataset 50k (FSD50k) (Fonseca et al., 2021). This dataset, available in both 16 kHz and 44.1 kHz sampling rates, comes with time-stamped annotations for each class: genre for music, audio-tags for sound effects, and transcription for speech.

The creation process of DnR was centered on addressing class overlap and relative source levels in the mix within a single-channel⁴ context. It includes four categories: speech, music, foreground effects, and background effects – the latter two being merged into a single submix. All mixtures have a duration of 60 seconds, encompassing multiple full speech utterances and sufficient onsets and offsets between classes. File count for each class was set via a zero-truncated Poisson distribution, and relative amplitude levels across the classes were determined per industry standards and prior studies as discussed by Petermann et al. (2022). Each sound file’s gain was individually adjusted to add variability while preserving realistic consistency across the mix. The final dataset, divided into training, validation, and testing subsets in line with base dataset proportions, comprises 3,406 training mixtures, 487 validation mixtures, and 973 test mixtures.

While the DnR dataset took care to simulate realistic cinematic mixtures, there are some notable differences between the source material used to create DnR and actual cinematic audio:

• Read speech vs. emotional speech – First, LibriSpeech contains read speech from audio books, which may have significant timbral differences compared to the emotional speech typically used by film actors.
• Vocals in music stems – Second, many of the musical genres from the FMA dataset contain vocals. While music with vocals is used in cinema, the majority of cinematic music does not contain singing. Thus, the prevalence of music with vocals may be overrepresented in FMA compared to the hidden test data.
• Production quality – Finally, Librispeech, FMA, and FSD50K are all crowd-sourced datasets, and there may be significant differences in terms of recording hardware and post-production effects compared to actual movies. We will investigate this in more detail in Section 5.

In summary, it is expected that mismatches such as this may limit performance of separation models trained only on DnR.

For Leaderboard A, participants were required to only utilize the training and validation split of the DnR dataset in training their systems.

3.2 CDXDB23 – Hidden Test Dataset

To rank the submissions, we generated a novel dataset derived from authentic Sony Pictures movies and we will refer in the following to this dataset as cinematic demixing database (CDXDB23). It comprises 11 movies with a total of 156 clips each with an average length of 11 seconds, amounting overall to approximately 28.7 minutes of content. The audio was originally at a higher sample rate, but we downsampled it to 44.1 kHz stereo to match the sample rate of the DnR dataset. This was done to avoid requiring participants in Leaderboard A to design systems that can upscale to a higher sampling rate. Figure 1 shows the distribution of genres and release years of the eleven movies in CDXDB23. Please note that a single movie can fall under multiple genres, such as Animation and Family. This characteristic is reflected in the bar plot, where the representation of movies in various genres contributes to the observed distribution. From Figure 1 we can observe that they are recent movies covering a wide variety of genres.

The original data supplied by Sony Pictures was formatted as 5.1-channel, 48 kHz, and 24-bit with several stem tracks for each movie, encompassing either dialogue, effects, music, or their combination. We manually annotated the sound events with one class label (dialogue, sound effects, or music) within each stem and carefully selected segments to ensure a balanced representation of each class in the resulting mixture. Specifically, to exclude extremely low-amplitude sound sources, we computed for each class the root mean square amplitude $RMSj=(1NΣn=1N‖sj(n)‖2)12$M16 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[{\rm{RM}}{{\rm{S}}_j} = {({\textstyle{1 \over N}}\Sigma _{n = 1}^N\Vert {{\bf{s}}_j}(n)\Vert^{2})^{{\textstyle{1 \over 2}}}}\] \end{document} and excluded segments where this value was below a threshold τ_j for any j ∈ {DX,FX,MX}. Empirically, we found the thresholds

to give good test samples. On occasion, environmental noise was unintentionally recorded, or dialogue/vocal components appeared in the effects or music stems. We made diligent efforts to minimize the inclusion of such samples by manually inspecting all data.

Please note that we are unable to provide further details about the movies (e.g., title or actors) to prevent participants in a future challenge from fine-tuning their models based on this specific information. However, we made available demo samples from “Kilian’s Game”, a short film produced by Sony Pictures to demonstrate the latest filmmaking technologies. These samples could be used by participants to test their submissions and to see the performance on real movie audio.⁵ The samples from “Kilian’s Game” were not used to rank the submissions.

Ideally, we should also use authentic movie data for training models. During the preparation of CDXDB23, we noticed that this is actually not straightforward. One problem is the preparation of the three-way stems from movie audio, which is a time-intensive process. The material is not readily accessible, requiring reloading all raw tracks into the Digital Audio Workstation (DAW), deciding for each of them the class it belongs to, and finally bouncing the stems for each class. Additionally, we noticed the challenge of other sound classes infiltrating a single stem. For example, dialogue stems can contain sound effects recorded on-stage. Consequently, after bouncing the stems, one has to manually annotate all audio material to find suitable time regions for the three-way separation, leading to a smaller dataset suitable only for testing, as exemplified by CDXDB23.

3.3 Cocktail-Fork Baseline

As part of the challenge, MERL open-sourced their multi-resolution CrossNet (MRX) (Petermann et al., 2022), an improved version of CrossNet-UMX (X-UMX) (Sawata et al., 2021, 2023), which itself is an improved version of Open-Unmix (UMX) (Stöter et al., 2019). MRX leverages multiple short-time Fourier transform (STFT) resolutions of the mixture, enhancing the estimation process as it allows to better address the variety of acoustic characteristics of the three source types. The entire system is available on GitHub.⁶

Using the available pre-trained model on DnR, a baseline submission was created and made available for all participants as part of the starter-kit.⁷ We noticed already during the preparation of the baseline that scaling the input mixture is beneficial and, hence, apply the scaling

i.e., the cocktail-fork model is run on the peak normalized mixture. Training of MRX utilized scale-invariant signal-to-distortion ratio (SI-SDR) loss, necessitating subsequent scale estimation using least-squares according to the formula

for any j ∈ {DX,FX,MX}. Furthermore, a post-processing step was implemented to ensure mixture consistency (Wisdom et al., 2019), where we first compute the residual $r(n)=x(n)−s^DX(n)−s^FX(n)−s^MX(n)$M17 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[{\bf{r}}(n) = {\bf{x}}(n) - {{\bf{\hat s}}_{{\rm{DX}}}}(n) - {{\bf{\hat s}}_{{\rm{FX}}}}(n) - {{\bf{\hat s}}_{{\rm{MX}}}}(n)\] \end{document} which is then distributed to the estimates

This post-processing was beneficial as the residual contains mostly sound effects and background music. SDRFX improved by +1.1 dB and SDRMX by +0.7 dB, resulting in an overall improvement of +0.6 dB. The performance of the cocktail-fork baseline on CDXDB23 can be found in Table 1.

After the challenge, we revisited this baseline as many participants recognized a distribution mismatch between DnR and CDXDB23, which can also be seen in Table 1 in the lower scores of this model. In Section 5.2, we will present two new versions of the cocktail-fork baseline with improved performance due to adjusting the loudness or equalization of DnR during training.

4.1 Team JusperLee (Kai Li, Yi Luo, Jianwei Yu, Rongzhi Gu)

Final ranking: Leaderboard A: —, Leaderboard B: 1^st

4.2 Team ZFTurbo (Roman Solovyev, Alexander Stempkovskiy, Tatiana Habruseva)

Final ranking: Leaderboard A: —, Leaderboard B: 2^nd

4.3 Team mp3d (Mikhail Sukhovei)

Final ranking: Leaderboard A: 2^nd, Leaderboard B: 5^th

4.3.2 Results

During the competition, we observed that validation on DnR is significantly different from metrics obtained on CDXDB23. This difference arises due to the dependence of the MRX and MRX-C model performance on the volume of the input signal. For testing on DnR, the optimal value was found to be –27 LUFS, which yielded the maximum SDR value. To obtain the optimal input volume for real world data, we propose a Realistic Evaluation Dataset (RED) consisting of 26 stereo audio tracks of 20 seconds each. These audio samples were manually compiled, with an average sample volume of approximately –15 LUFS and a sampling rate of 44.1 kHz. Additionally, each audio file includes separate tracks for dialogue, music, and effects, all with the same duration and with average sample volumes of –24.4 ± 4.5 LUFS for dialogues, –18.8 ± 0.5 LUFS for effects and –18.4 ± 5.0 LUFS for music. All the original audio files are sourced from the open archive.org platform. The RED dataset was used only to select the optimal value of the input signal volume of the model.

As shown in Figure 5, the optimal volume value differed significantly from that of DnR, both in the case of RED and CDXDB23. Furthermore, the SDR metric on RED was more consistent with CDXDB23. After separating the sources, they were brought back to the original volume and a Wiener filter and post-processing scaling were applied. Post-processing scaling involved multiplying the estimated sources by a factor of $1α$M19 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[{\textstyle{1 \over \alpha }}\] \end{document} where $α=Σnx(n)Ts^(n)10−7+Σn‖s^(n)‖2,x(n)$M20 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[\alpha = {\textstyle{{{\Sigma _n}{\bf{x}}{{(n)}^T}{\bf{\hat s}}(n)} \over {{{10}^{ - 7}} + {\Sigma _n}{\bf{\hat s}}({\bf{n}}){^2}}}},{\bf{x}}(n)\] \end{document} , is the mixture and $s^(n)$M21 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[{\bf{\hat s}}(n)\] \end{document} is an estimated source.

Figure 5 

SDR dependencies on the input volume in LUFS for music, dialogue, and effects. A solid line shows SDR values on RED; crosses mark SDR on CDXDB23. Horizontal dashed and dotted lines show SDR for models without converting the volume of the input signal. The MRX model is blue, MRX-C is orange, MRX-C with a Wiener filter is green, and MRX-C with post-processing scaling is red. In the case of testing MRX-C scaling on the CDXDB23, the SDR values are only available for effects (Team mp3d).

Table 5 summarizes the SDR metrics on RED for the baseline MRX solution, MRX-C, MRX-C with Wiener filter, and MRX-C with source scaling. As shown in the table, the MRX-C model yields a 0.1 dB improvement in dialogue and a 0.1 dB decrease in effects. The Wiener filter yields a 0.1 dB improvement in music, a 0.3 dB improvement in dialogue, and a 0.3 dB improvement in effects. Post-processing scaling results in a 0.2 dB decrease in music, a 0.6 dB decrease in dialogue, and a 0.3 dB increase in effects. Our final solution involved applying the Wiener filter only to the dialogue and post-processing scaling only to the effects.

Table 5

SDR values obtained during testing on RED for MRX, MRX-C, MRX-C with Wiener filter, and MRX-C with scaling. SDR values from the table are maximum possible values from all input volumes (Team mp3d).

Model	Global SDR (dB)
	Mean	Dialogue	Effects	Music
MRX	4.38	8.38	1.72	3.02
MRX-C	4.36	8.48	1.62	2.99
MRX-C Wiener	4.57	8.75	1.90	3.07
MRX-C scaling	4.24	7.92	1.95	2.85

5.1 Difference in Signal Statistics

First, we will compare the signal characteristics between DnR and CDXDB23. Our focus will be on loudness, equalization, stereo panning, and dynamic range compression as they are key elements in audio mixing (Martínez-Ramírez et al., 2022).¹⁸

Loudness—We measured the loudness for each audio clip in both datasets using ITU-R BS.1770-4 (International Telecommunications Union, 2015) with the help of pyloudnorm (Steinmetz and Reiss, 2021). The average loudness values are shown in Table 6 and the histograms can be found in Figure 6. We can observe that CDXDB23 utilizes the full range of loudness for all three classes whereas DnR has a more limited range. On average, DnR is 4 LUFS louder than CDXDB23. Notably, CDXDB23 uses the same loudness level for effects and music which is 5 LUFS lower than dialogue. This balance is likely due to a post-production step which was not considered in DnR.

Equalization—To assess equalization differences, we normalized each waveform to –24 LUFS and calculated the magnitude STFT spectrogram using a Hann window of 4096 samples with 75% overlap. The average equalization curves are displayed in Figure 7. It shows that CDXDB23 generally has a faster decay at low and high frequencies but more energy in the mid-frequency range compared to DnR. We attribute this to the use of parametric EQs containing low and high shelf filters in the post production process for CDXDB23, while DnR consisted mostly of web content, which likely lacked professional post production.

Amplitude panning—We calculated the Stereo Panning Spectrum as outlined by Tzanetakis et al. (2007); Avendano (2003) and as used by Martínez-Ramírez et al. (2022). Using the magnitude spectrogram from the earlier equalization analysis, we computed

where X_L/R(f) denotes the left/right channel magnitude spectrogram. $Ψ(f)$M22 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[\Psi (f)\] \end{document} measures whether frequencies are panned, regardless of direction, while Δ(f) measures which direction frequencies are panned to. Figures 8 and 9 show the average values for both datasets. It is important to note that DnR is monaural, which results in a horizontal line for $Ψ(f)$M23 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[\Psi (f)\] \end{document} and Δ(f). From these figures, we see that in the audio mixing process, dialogue is typically centered, while effects are more often panned to one side. Music shows the most varied panning. Figure 9 indicates that there is no specific preferred direction for panning in the datasets.

Dynamic range compression (DRC)—Lastly, we analyzed the DRC by calculating the average peak value as DRC usually alters the transients. We started by normalizing the loudness of the audio waveform to –24 LUFS. Then, we used the high frequency content (HFC) method for onset detection, as described by Masri (1996); Brossier et al. (2019) and as implemented by Martínez-Ramírez et al. (2022). For each audio clip, we calculated the average peak level $Pμ$M34 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[{{\cal P}_\mu }\] \end{document} . This measure helps us understand the extent of DRC applied; larger $Pμ$M35 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[{{\cal P}_\mu }\] \end{document} values indicate less compression since the peaks are more pronounced at the same loudness level. From the data in Table 6, we see that CDXDB23 exhibits more uniform compression compared to DnR. Notably, in DnR, effects are less compressed compared to dialogue and music. This contrast in compression is not observed in CDXDB23, where compression is more consistently applied across all three classes due to being professionally produced.

5.2 Improving the Cocktail-Fork Baseline

Using our understanding of the distribution differences from the previous section, we aim to enhance the cocktail-fork model from Section 3.3. We created two updated versions of DnR, modifying either the average loudness or the average equalization for each audio source. For the loudness adjustment, we changed the loudness of each source stem by a specific amount. For example, we altered the loudness of each dialogue stem by 4 LUFS (from –28.4 to –24.4 LUFS). This adjustment was made so that the average loudness of the modified DnR matches that of CDXDB23. Regarding equalization, we designed a 101-tap FIR filter for each audio source. The magnitude response of this filter is the square root of the difference between the average equalization of CDXDB23 and DnR. We then applied this filter to each stem using forward-backward filtering (filtfilt), as mentioned by Martínez-Ramírez et al. (2022). This process modifies the amplitude without altering the phase of the audio.

Table 7 shows the improvements to the cocktail-fork model. Please note that an additional feature for mixture loudness normalization was introduced with version 1.1, setting the mixture to –27 LUFS. We will discuss the results without considering this normalization, although similar trends are observed with it. The results in Table 7 indicate that adjusting the loudness is particularly effective. The mean SDR improved from –0.1 dB to 1.3 dB, primarily due to enhanced performance in dialogue. Adjusting the equalization also showed benefits, with an overall improvement of 0.3 dB. Here, both dialogue and effects showed improvement, but there was a slight decrease of 0.2 dB in music. We believe this decrease in music is linked to the fluctuation in the equalization curve shown in Figure 7, which correlates with the frequencies of musical notes. This fluctuation could inadvertently act as a marker on the music stems, making them easier for the model to separate, leading to a slight decline in music performance. Both trained models are available on the cocktail-fork GitHub⁶.

In summary, by aligning the DnR dataset more closely with a more realistic dataset like CDXDB23, we significantly enhanced the performance of the model. This approach presents a promising avenue for future research in this field. Besides using mono-to-stereo augmentation for stereo panning and compressors to adjust the DRC, also combining all of them should be considered to close the distribution mismatch as much as possible. Using the reported statistics in Section 5.1 will help to choose realistic parameters for the data augmentation.