[big refactoring] Audio latency fix for Android. Support to preload effects on Android now. (#15875)
* Audio latency fix for Android. Support preload effects on Android now.
Squashed commits:
[b6d80fe] log fix
[a0a918e] Fixes assetFd didn't be released while PcmData is returned from cache.
[4b956ba] Potential crash fix for PcmAudioPlayer while pause / resume.
[398ab8c] Updates LOG_TAG position in AudioEngine-inl.cpp
[e3634e7] include stdlib.h for posix_memalign
[9004074] fixes setVolume logical error.
[c96df46] Don't use another thread for mixing, enqueue is in a seperated thread, therefore doing mixing in another thread will waste more time.
[0a4c1a8] Adds setLoop, setVolume, setPostion support for Track
[c35fb20] Fixed include.
[cdd9d32] Do mixing by ourself. (TO BE POLISHED)
[6447025] µ -> u since µ could not be shown on some android devices.
[97be0c6] Don't send a silence clip.
[c1607ed] Make linter.py happy.
[0898b54] Puts enqueue & SetPlayState in PcmAudioPlayer::play to thread pool.
[b79fc01] Adds getDuration, getPosition support for PcmAudioPlayer
[80fa2ab] minor fix of the code position of resetting state to State::INITIALIZED
[d9c62f1] underrun fix for PcmAudioPlayer.
[9c2212a] UrlAudioPlayer, playOverMutex should be static, and should be used in update method.
[1519d2e] static variables
[19da936] _pcmAudioPlayer Null pointer check in AudioPlayerProvider.
[e6b0d14] Updates audio performance test.
[fc01dd4] Registers foreground & background event in AudioEngine-inl.cpp(android), the callback should invoke `provider`'s pause & resume method.
[e00a886] TBD: Pause & resume support for PcmAudioPlayerPool.
Since OpenSLES audio resources are expensive and device shared, we should delete all unused PcmAudioPlayers in pool while pause and re-create them while resume.
But this commit isn't finished yet, I don't find a better way to register pause&resume event in AudioEngine module.
[9e42ea3] Interleave mono audio to stereo audio. PcmAudioPlayerPool only contains PcmAudioPlayers with 2 channels.
[3f18d05] Adds a strategy for checking small size of different file formats.
[753ff49] Adds performance test for AudioEngine.
[09d3045] Releases an extra PcmAudioPlayer for UrlAudioPlayer while allocating PcmAudioPlayer fails.
[9dd4477] Using std::move for PcmData move constructor & move assignment.
[6ca3bcb] some fixes:
1) new -> new (std::nothrow)
2) break if allocate PcmAudioPlayer fails
3) renames 'initForPlayPcmData' to 'init'
4) PcmAudioPlayer destructor deadlock if 'init' failed
[54675b6] include path fix.
[a1903ca] More refactorings.
[19b9498] Makes linter.py happy. :)
[923c530] Fixes:
1) Avoid getFileInfo to be invoked twice
2) A critical bug fix for UrlAudioPlayer and adds detailed comments
3) __clang__ compiler option fix for AudioResamplerSinc.cpp.
[5ec4faf] minor fix.
[faaa0f3] output a log in the destructor of UrlAudioPlayer.
[9c20355] NewAudioEngineTest,TestControll crash fix.
[f114464] fixes an unused import.
[1dc5dab] Better algorithm for allocating PcmAudioPlayer.
[331a213] minor fix.
[e54084a] null -> nullptr
[f9a0389] Support uncache.
[89a364f] Removes unused update, and TODO uncache functionality.
[1732bf9] Supports AudioEngineImpl::setFinishCallback for android.
[43d1596] UrlAudioPlayer::stop fix.
[e2ee941] Test case fix in NewAudioEngineTest/AudioIssue11143Test
[5c5ba01] More fixes for making cpp-tests/New Audio Engine Test happy.
[8b554a3] Adds log while remove player from map.
[ed71322] If original file is larger than 30k bytes, consider it's a large audio file.
[fb1845a] Updates project.properties
[6f3839f] minor log output fix in AudioEngine-inl.cpp
[c68bc6c] Don't resample if the sample rate of the decoded pcm data matchs the device's.
[43ca45f] PcmAudioPlayers also need to be removed while they play over, but should not be deleted since their lifecycle is managed by PcmAudioPlayerPool.
[f5e63c9] Audio latency fix for Android. Support preload effects on Android now.
* Supports to loading audio files asynchronously.
* Crash fix for stop audio right after play2d.
* Minor fix for logic in AudioMixerController.cpp
* Adds missing files (CCThreadPool.h/.cpp).
* Minor fix for including.
* Minor fix for missing include <functional> in Track.h
* update license information in audio.h
* Don't use std::future/std::promise anymore since ndk counldn't support it well in armeabi arch.
* isSmallFile postion updated, fixes large audio file goto the checking logic of cache.
* std::atomic<int> isn't supported by ndk-r10e while compiling with `armeabi` arch, using a int with a mutex instead.
* fixes __isnanf & posix_memalign doesn't exist on low api (<=16) devices.
* namespace updated: cocos2d -> cocos2d::experimental
* Removes commented code in AudioMixerController.h/.cpp
* Removes unused code again, and fixes a memory leak of `Track` instance.
* Oops, namespace changed.
* Only outputs log in debug mode.
* Uses ALOGV for outputing logs in AudioEngine-inl.cpp
* const PcmData& -> PcmData for Track
* Fixes a protential crash in NewAudioEngineTest
* Adds `COCOS` prefix in header #ifndef COCOS_BALABALA #define COCOS_BALABALA
* Uses _ prefix for cocos code style instead of `m` prefix.
* Deletes AudioResamplerSinc related files.
* Bug fix from @minggo's reply on github.
* Don't need to invoke pause after in UrlAudioPlayer::prepare.
* Updates ThreadPool class, uses enum class and adds const keyword.
2016-07-18 10:22:40 +08:00
|
|
|
/*
|
|
|
|
* Copyright (C) 2007 The Android Open Source Project
|
|
|
|
*
|
|
|
|
* Licensed under the Apache License, Version 2.0 (the "License");
|
|
|
|
* you may not use this file except in compliance with the License.
|
|
|
|
* You may obtain a copy of the License at
|
|
|
|
*
|
|
|
|
* http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
*
|
|
|
|
* Unless required by applicable law or agreed to in writing, software
|
|
|
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
* See the License for the specific language governing permissions and
|
|
|
|
* limitations under the License.
|
|
|
|
*/
|
|
|
|
|
|
|
|
#pragma once
|
|
|
|
|
|
|
|
#include <stdint.h>
|
|
|
|
#include <sys/types.h>
|
|
|
|
#include <android/log.h>
|
|
|
|
#include <sys/system_properties.h>
|
|
|
|
|
|
|
|
#include "audio/android/AudioBufferProvider.h"
|
|
|
|
|
|
|
|
//#include <cutils/compiler.h>
|
|
|
|
//#include <utils/Compat.h>
|
|
|
|
|
|
|
|
//#include <media/AudioBufferProvider.h>
|
|
|
|
//#include <system/audio.h>
|
|
|
|
#include <assert.h>
|
|
|
|
#include "audio/android/audio.h"
|
|
|
|
|
2019-10-23 14:58:31 +08:00
|
|
|
namespace cocos2d {
|
[big refactoring] Audio latency fix for Android. Support to preload effects on Android now. (#15875)
* Audio latency fix for Android. Support preload effects on Android now.
Squashed commits:
[b6d80fe] log fix
[a0a918e] Fixes assetFd didn't be released while PcmData is returned from cache.
[4b956ba] Potential crash fix for PcmAudioPlayer while pause / resume.
[398ab8c] Updates LOG_TAG position in AudioEngine-inl.cpp
[e3634e7] include stdlib.h for posix_memalign
[9004074] fixes setVolume logical error.
[c96df46] Don't use another thread for mixing, enqueue is in a seperated thread, therefore doing mixing in another thread will waste more time.
[0a4c1a8] Adds setLoop, setVolume, setPostion support for Track
[c35fb20] Fixed include.
[cdd9d32] Do mixing by ourself. (TO BE POLISHED)
[6447025] µ -> u since µ could not be shown on some android devices.
[97be0c6] Don't send a silence clip.
[c1607ed] Make linter.py happy.
[0898b54] Puts enqueue & SetPlayState in PcmAudioPlayer::play to thread pool.
[b79fc01] Adds getDuration, getPosition support for PcmAudioPlayer
[80fa2ab] minor fix of the code position of resetting state to State::INITIALIZED
[d9c62f1] underrun fix for PcmAudioPlayer.
[9c2212a] UrlAudioPlayer, playOverMutex should be static, and should be used in update method.
[1519d2e] static variables
[19da936] _pcmAudioPlayer Null pointer check in AudioPlayerProvider.
[e6b0d14] Updates audio performance test.
[fc01dd4] Registers foreground & background event in AudioEngine-inl.cpp(android), the callback should invoke `provider`'s pause & resume method.
[e00a886] TBD: Pause & resume support for PcmAudioPlayerPool.
Since OpenSLES audio resources are expensive and device shared, we should delete all unused PcmAudioPlayers in pool while pause and re-create them while resume.
But this commit isn't finished yet, I don't find a better way to register pause&resume event in AudioEngine module.
[9e42ea3] Interleave mono audio to stereo audio. PcmAudioPlayerPool only contains PcmAudioPlayers with 2 channels.
[3f18d05] Adds a strategy for checking small size of different file formats.
[753ff49] Adds performance test for AudioEngine.
[09d3045] Releases an extra PcmAudioPlayer for UrlAudioPlayer while allocating PcmAudioPlayer fails.
[9dd4477] Using std::move for PcmData move constructor & move assignment.
[6ca3bcb] some fixes:
1) new -> new (std::nothrow)
2) break if allocate PcmAudioPlayer fails
3) renames 'initForPlayPcmData' to 'init'
4) PcmAudioPlayer destructor deadlock if 'init' failed
[54675b6] include path fix.
[a1903ca] More refactorings.
[19b9498] Makes linter.py happy. :)
[923c530] Fixes:
1) Avoid getFileInfo to be invoked twice
2) A critical bug fix for UrlAudioPlayer and adds detailed comments
3) __clang__ compiler option fix for AudioResamplerSinc.cpp.
[5ec4faf] minor fix.
[faaa0f3] output a log in the destructor of UrlAudioPlayer.
[9c20355] NewAudioEngineTest,TestControll crash fix.
[f114464] fixes an unused import.
[1dc5dab] Better algorithm for allocating PcmAudioPlayer.
[331a213] minor fix.
[e54084a] null -> nullptr
[f9a0389] Support uncache.
[89a364f] Removes unused update, and TODO uncache functionality.
[1732bf9] Supports AudioEngineImpl::setFinishCallback for android.
[43d1596] UrlAudioPlayer::stop fix.
[e2ee941] Test case fix in NewAudioEngineTest/AudioIssue11143Test
[5c5ba01] More fixes for making cpp-tests/New Audio Engine Test happy.
[8b554a3] Adds log while remove player from map.
[ed71322] If original file is larger than 30k bytes, consider it's a large audio file.
[fb1845a] Updates project.properties
[6f3839f] minor log output fix in AudioEngine-inl.cpp
[c68bc6c] Don't resample if the sample rate of the decoded pcm data matchs the device's.
[43ca45f] PcmAudioPlayers also need to be removed while they play over, but should not be deleted since their lifecycle is managed by PcmAudioPlayerPool.
[f5e63c9] Audio latency fix for Android. Support preload effects on Android now.
* Supports to loading audio files asynchronously.
* Crash fix for stop audio right after play2d.
* Minor fix for logic in AudioMixerController.cpp
* Adds missing files (CCThreadPool.h/.cpp).
* Minor fix for including.
* Minor fix for missing include <functional> in Track.h
* update license information in audio.h
* Don't use std::future/std::promise anymore since ndk counldn't support it well in armeabi arch.
* isSmallFile postion updated, fixes large audio file goto the checking logic of cache.
* std::atomic<int> isn't supported by ndk-r10e while compiling with `armeabi` arch, using a int with a mutex instead.
* fixes __isnanf & posix_memalign doesn't exist on low api (<=16) devices.
* namespace updated: cocos2d -> cocos2d::experimental
* Removes commented code in AudioMixerController.h/.cpp
* Removes unused code again, and fixes a memory leak of `Track` instance.
* Oops, namespace changed.
* Only outputs log in debug mode.
* Uses ALOGV for outputing logs in AudioEngine-inl.cpp
* const PcmData& -> PcmData for Track
* Fixes a protential crash in NewAudioEngineTest
* Adds `COCOS` prefix in header #ifndef COCOS_BALABALA #define COCOS_BALABALA
* Uses _ prefix for cocos code style instead of `m` prefix.
* Deletes AudioResamplerSinc related files.
* Bug fix from @minggo's reply on github.
* Don't need to invoke pause after in UrlAudioPlayer::prepare.
* Updates ThreadPool class, uses enum class and adds const keyword.
2016-07-18 10:22:40 +08:00
|
|
|
|
|
|
|
|
|
|
|
class AudioResampler {
|
|
|
|
public:
|
|
|
|
// Determines quality of SRC.
|
|
|
|
// LOW_QUALITY: linear interpolator (1st order)
|
|
|
|
// MED_QUALITY: cubic interpolator (3rd order)
|
|
|
|
// HIGH_QUALITY: fixed multi-tap FIR (e.g. 48KHz->44.1KHz)
|
|
|
|
// NOTE: high quality SRC will only be supported for
|
|
|
|
// certain fixed rate conversions. Sample rate cannot be
|
|
|
|
// changed dynamically.
|
|
|
|
enum src_quality {
|
|
|
|
DEFAULT_QUALITY=0,
|
|
|
|
LOW_QUALITY=1,
|
|
|
|
MED_QUALITY=2,
|
|
|
|
HIGH_QUALITY=3,
|
|
|
|
VERY_HIGH_QUALITY=4,
|
|
|
|
};
|
|
|
|
|
|
|
|
static const CONSTEXPR float UNITY_GAIN_FLOAT = 1.0f;
|
|
|
|
|
|
|
|
static AudioResampler* create(audio_format_t format, int inChannelCount,
|
|
|
|
int32_t sampleRate, src_quality quality=DEFAULT_QUALITY);
|
|
|
|
|
|
|
|
virtual ~AudioResampler();
|
|
|
|
|
|
|
|
virtual void init() = 0;
|
|
|
|
virtual void setSampleRate(int32_t inSampleRate);
|
|
|
|
virtual void setVolume(float left, float right);
|
|
|
|
virtual void setLocalTimeFreq(uint64_t freq);
|
|
|
|
|
|
|
|
// set the PTS of the next buffer output by the resampler
|
|
|
|
virtual void setPTS(int64_t pts);
|
|
|
|
|
|
|
|
// Resample int16_t samples from provider and accumulate into 'out'.
|
|
|
|
// A mono provider delivers a sequence of samples.
|
|
|
|
// A stereo provider delivers a sequence of interleaved pairs of samples.
|
|
|
|
//
|
|
|
|
// In either case, 'out' holds interleaved pairs of fixed-point Q4.27.
|
|
|
|
// That is, for a mono provider, there is an implicit up-channeling.
|
|
|
|
// Since this method accumulates, the caller is responsible for clearing 'out' initially.
|
|
|
|
//
|
|
|
|
// For a float resampler, 'out' holds interleaved pairs of float samples.
|
|
|
|
//
|
|
|
|
// Multichannel interleaved frames for n > 2 is supported for quality DYN_LOW_QUALITY,
|
|
|
|
// DYN_MED_QUALITY, and DYN_HIGH_QUALITY.
|
|
|
|
//
|
|
|
|
// Returns the number of frames resampled into the out buffer.
|
|
|
|
virtual size_t resample(int32_t* out, size_t outFrameCount,
|
|
|
|
AudioBufferProvider* provider) = 0;
|
|
|
|
|
|
|
|
virtual void reset();
|
|
|
|
virtual size_t getUnreleasedFrames() const { return mInputIndex; }
|
|
|
|
|
|
|
|
// called from destructor, so must not be virtual
|
|
|
|
src_quality getQuality() const { return mQuality; }
|
|
|
|
|
|
|
|
protected:
|
|
|
|
// number of bits for phase fraction - 30 bits allows nearly 2x downsampling
|
|
|
|
static const int kNumPhaseBits = 30;
|
|
|
|
|
|
|
|
// phase mask for fraction
|
|
|
|
static const uint32_t kPhaseMask = (1LU<<kNumPhaseBits)-1;
|
|
|
|
|
|
|
|
// multiplier to calculate fixed point phase increment
|
|
|
|
static const double kPhaseMultiplier;
|
|
|
|
|
|
|
|
AudioResampler(int inChannelCount, int32_t sampleRate, src_quality quality);
|
|
|
|
|
|
|
|
// prevent copying
|
|
|
|
AudioResampler(const AudioResampler&);
|
|
|
|
AudioResampler& operator=(const AudioResampler&);
|
|
|
|
|
|
|
|
int64_t calculateOutputPTS(int outputFrameIndex);
|
|
|
|
|
|
|
|
const int32_t mChannelCount;
|
|
|
|
const int32_t mSampleRate;
|
|
|
|
int32_t mInSampleRate;
|
|
|
|
AudioBufferProvider::Buffer mBuffer;
|
|
|
|
union {
|
|
|
|
int16_t mVolume[2];
|
|
|
|
uint32_t mVolumeRL;
|
|
|
|
};
|
|
|
|
int16_t mTargetVolume[2];
|
|
|
|
size_t mInputIndex;
|
|
|
|
int32_t mPhaseIncrement;
|
|
|
|
uint32_t mPhaseFraction;
|
|
|
|
uint64_t mLocalTimeFreq;
|
|
|
|
int64_t mPTS;
|
|
|
|
|
|
|
|
// returns the inFrameCount required to generate outFrameCount frames.
|
|
|
|
//
|
|
|
|
// Placed here to be a consistent for all resamplers.
|
|
|
|
//
|
|
|
|
// Right now, we use the upper bound without regards to the current state of the
|
|
|
|
// input buffer using integer arithmetic, as follows:
|
|
|
|
//
|
|
|
|
// (static_cast<uint64_t>(outFrameCount)*mInSampleRate + (mSampleRate - 1))/mSampleRate;
|
|
|
|
//
|
|
|
|
// The double precision equivalent (float may not be precise enough):
|
|
|
|
// ceil(static_cast<double>(outFrameCount) * mInSampleRate / mSampleRate);
|
|
|
|
//
|
|
|
|
// this relies on the fact that the mPhaseIncrement is rounded down from
|
|
|
|
// #phases * mInSampleRate/mSampleRate and the fact that Sum(Floor(x)) <= Floor(Sum(x)).
|
|
|
|
// http://www.proofwiki.org/wiki/Sum_of_Floors_Not_Greater_Than_Floor_of_Sums
|
|
|
|
//
|
|
|
|
// (so long as double precision is computed accurately enough to be considered
|
|
|
|
// greater than or equal to the Floor(x) value in int32_t arithmetic; thus this
|
|
|
|
// will not necessarily hold for floats).
|
|
|
|
//
|
|
|
|
// TODO:
|
|
|
|
// Greater accuracy and a tight bound is obtained by:
|
|
|
|
// 1) subtract and adjust for the current state of the AudioBufferProvider buffer.
|
|
|
|
// 2) using the exact integer formula where (ignoring 64b casting)
|
|
|
|
// inFrameCount = (mPhaseIncrement * (outFrameCount - 1) + mPhaseFraction) / phaseWrapLimit;
|
|
|
|
// phaseWrapLimit is the wraparound (1 << kNumPhaseBits), if not specified explicitly.
|
|
|
|
//
|
|
|
|
inline size_t getInFrameCountRequired(size_t outFrameCount) {
|
|
|
|
return (static_cast<uint64_t>(outFrameCount)*mInSampleRate
|
|
|
|
+ (mSampleRate - 1))/mSampleRate;
|
|
|
|
}
|
|
|
|
|
|
|
|
inline float clampFloatVol(float volume) {
|
|
|
|
if (volume > UNITY_GAIN_FLOAT) {
|
|
|
|
return UNITY_GAIN_FLOAT;
|
|
|
|
} else if (volume >= 0.) {
|
|
|
|
return volume;
|
|
|
|
}
|
|
|
|
return 0.; // NaN or negative volume maps to 0.
|
|
|
|
}
|
|
|
|
|
|
|
|
private:
|
|
|
|
const src_quality mQuality;
|
|
|
|
|
|
|
|
// Return 'true' if the quality level is supported without explicit request
|
|
|
|
static bool qualityIsSupported(src_quality quality);
|
|
|
|
|
|
|
|
// For pthread_once()
|
|
|
|
static void init_routine();
|
|
|
|
|
|
|
|
// Return the estimated CPU load for specific resampler in MHz.
|
|
|
|
// The absolute number is irrelevant, it's the relative values that matter.
|
|
|
|
static uint32_t qualityMHz(src_quality quality);
|
|
|
|
};
|
|
|
|
|
|
|
|
// ----------------------------------------------------------------------------
|
2019-10-23 14:58:31 +08:00
|
|
|
} // namespace cocos2d {
|