Testing Blog

WebRTC Audio Quality Testing

Friday, November 08, 2013
Share on Twitter Share on Facebook
Google
Labels: Patrik Höglund , WebRTC

16 comments :

  1. man9ar00November 8, 2013 at 5:56:00 PM PST

    Nice post. I think this may have use for SDET/QA beyond streaming audio/media space. The technique may prove useful in telecom hardware/software/services testing as well.

    ReplyDelete
    Replies
      Reply
  2. BloodyArmyJanuary 6, 2014 at 8:00:00 PM PST

    Great Post! Thanks for sharing. :)

    Just some queries on the big picture here:- this looks like a one off project and the challenge here is considered quite unique and not the usual day to day question that we encounter during testing. It interests me when the team manage to find out so many metrics & formula to measure the quality of the application. Does it mean that the test team has also spent effort to understand the formula, code it into test for calculation, etc? Or the test team actually engage some SME (subject matter expert) to advise? Would hope to know how Google test team approach this situation? Besides, mind sharing how long did you guys use to complete this project?

    ReplyDelete
    Replies
      Reply
  3. UnknownJanuary 21, 2014 at 12:22:00 AM PST

    Great post, thanks!
    Is there any articles/posts analyzing WebRTC video quality? Many thanks.

    ReplyDelete
    Replies
      Reply
  4. UnknownJanuary 27, 2014 at 2:57:00 PM PST

    Great post! I managed to run the test, however, I am getting scores less or around 2. And yes, I used 'rebasing' technique with the right sampling rate, etc. I would like to try with your file and see if it makes it any better. Could you please share the wav file at pyauto_private/webrtc/human-voice-linux.wav since it seems we (public) doesnt seem to have access to that (or I couldnt figure out how). One thing to note is that I am trying this on a virtual machine (ubuntu flavor).

    Thanks!

    ReplyDelete
    Replies
    1. UnknownMay 28, 2014 at 9:59:00 AM PDT

      I'm a newbie to webrtc.
      Kindly, someone explain me in detail, how I can setup my machine to run the test? - I was able to compile the chrome nightly build using ninja. What should be the next step to run the above mentioned test (i.e., which file or command I should execute to perform the above mentioned tasks).

      Delete
      Replies
        Reply
    2. AnonymousAugust 26, 2014 at 2:38:00 AM PDT

      Ok, here's how you do it. First, get the code (http://dev.chromium.org/developers/how-tos/get-the-code) or update your existing checkout so you get the latest code (I landed some patches very recently). Instead of building chrome, build browser_tests. Configure your machine like instructed in https://code.google.com/p/chromium/codesearch#chromium/src/chrome/browser/media/chrome_webrtc_audio_quality_browsertest.cc&q=chrome_webrtc_a&sq=package:chromium&l=49 (if you run Mac you're out of luck; the test isn't implemented there).

      Then add this to the solutions list your .gclient file (it's in the folder above your chrome src/ folder:

      {
      "name" : "webrtc.DEPS",
      "url" : "svn://svn.chromium.org/chrome/trunk/deps/third_party/webrtc/webrtc.DEPS",
      "managed" : True,
      },

      That will download the resources you need. It will probably fail on downloading from the webrtc-chrome-resources bucket, so you need to comment that part out in the hooks in webrtc.DEPS/DEPS. Then you need to get a hold of PESQ (and if you're on windows: sox.exe), build those yourself and put them in src/chrome/test/data/webrtc/resources/tools. We can't redistribute those binaries but they're readily available on the web.

      Then run out/Debug/browser_tests --gtest_filter="WebRtcAudio*" --run-manual. If it works you'll get a PESQ score printed out.

      Good luck! :)

      Delete
      Replies
        Reply
    3. Reply
  5. AnonymousJanuary 28, 2014 at 1:16:00 AM PST

    Unknown: Sure, that file is private only for historical reasons. I'm going to try to pull it out to the public world when I get time. For now I'll attach it in comment #3 here: https://crbug.com/279195.

    The best way to figure out why the score is low is to listen to the recorded file (just uncomment the DeleteFile on trimmed_recording). Load it up in audacity and compare to the source file. Often you will find that the volume level of the recording is too low because the system's input or playback volume levels are wrong. If they're too low you'll get a bad recording and if they're too high you'll get distortions. Read the comments on the test on how to set up your machine carefully and look at the volume levels in pavucontrol. I think they're all 100% on our machines, but this may or may not be appropriate for your sound hardware.

    ReplyDelete
    Replies
    1. UnknownJanuary 28, 2014 at 2:20:00 PM PST

      Thanks for the answer Patrik - I think volume levels are okay. looking at the waveforms in audacity, recorded waveform is always expanded somewhat. To give an example, I shared a photo below where the original signal is 5.181 seconds long and recorded signal is 5.581 seconds long.

      https://plus.google.com/105460497673148445079/posts?banner=pwa

      From what I know, PESQ can deal with the silence in between talkspurts well if they are different between original and recorded files, i.e. one may want to reduce the silence period and play out quicker at the receiving side without much quality compromise. However, if my memory serves me right, PESQ cannot deal with the expanded wave forms, which could be the cause for low PESQ results in my environment.

      But this doesnt explain why your tests are at around 4. I wonder if virtual machine environment has a contributing factor here. Do you run your tests on physical machines? Thanks again.

      Delete
      Replies
        Reply
    2. Reply
  6. AnonymousJanuary 28, 2014 at 1:20:00 AM PST

    BloodyArmy: We do have very skilled audio engineers on the WebRTC team, which is how I learned about PESQ, sampling rates and so on. I wouldn't have been able to pull this test off without help from those experts for sure.

    I would say this test took about three manweeks to implement. It was very hard with all the OS-specific quirks, and it took a lot of testing and tweaking to get it to run well.

    ReplyDelete
    Replies
      Reply
  7. AnonymousJanuary 28, 2014 at 1:23:00 AM PST

    Akmal Nishanov: Yes! I did a tech talk on GTAC 2013. Find the video and slides here: http://www.youtube.com/watch?v=IbLNm3LsMaw&list=SPSIUOFhnxEiCODb8XQB-RUQ0RGNZ2yW7d

    There's no article like this one though; I might write one in the future.

    ReplyDelete
    Replies
      Reply
  8. continuousqaMarch 5, 2014 at 2:11:00 PM PST

    Hope this doesn't double post, tried posting and browser crashed... anyway, I found this article really appropriate to the webRTC work I'm testing. Very helpful. I saw in your future considerations you're looking at latency models. I did some work on that. Before webrtc, I was doing web automation, and I built a latency generator using a linux VM and Netem. I put squid on there to open a port and then route the browser calls through the proxy. Before the tests starts I would make a call that remotely sets the bandwidth/latency/packet loss profile on the VM. Then run the test through the VM. It worked very well for us. If you are interested I have a write up on it at my blog: http://sdet.us/simulating-real-world-latency-during-automation/

    ReplyDelete
    Replies
      Reply
  9. UnknownOctober 8, 2015 at 3:10:00 AM PDT

    Hi, Any progress for test under simulated network conditions which u listed in future work?

    ReplyDelete
    Replies
      Reply
  10. Vincent CJanuary 26, 2017 at 4:54:00 AM PST

    Where is the source code for chrome_webrtc_audio_quality_browsertest.cc? Has it moved, or been deleted from the chromium project?

    ReplyDelete
    Replies
    1. UnknownOctober 13, 2017 at 8:43:00 PM PDT

      The file name changed
      https://cs.chromium.org/chromium/src/chrome/browser/media/webrtc/webrtc_audio_quality_browsertest.cc?sq=package:chromium

      Delete
      Replies
        Reply
    2. Reply
  11. UnknownJune 1, 2017 at 3:31:00 AM PDT

    Thanks for sharing your honest experience. When I first took a look at my headshots,
    I wasn’t too thrilled with mine but you’ve given me a new perspective!

    ReplyDelete
    Replies
      Reply
  12. UnknownJanuary 23, 2018 at 4:15:00 AM PST

    What about some kind of fake device driver which makes a microphone-like device appear on the device level? Thanks for the post

    ReplyDelete
    Replies
      Reply
Add comment
Load more...

The comments you read and contribute here belong only to the person who posted them. We reserve the right to remove off-topic comments.

  

Labels


  • TotT 104
  • GTAC 61
  • James Whittaker 42
  • Misko Hevery 32
  • Code Health 31
  • Anthony Vallone 27
  • Patrick Copeland 23
  • Jobs 18
  • Andrew Trenk 13
  • C++ 11
  • Patrik Höglund 8
  • JavaScript 7
  • Allen Hutchison 6
  • George Pirocanac 6
  • Zhanyong Wan 6
  • Harry Robinson 5
  • Java 5
  • Julian Harty 5
  • Adam Bender 4
  • Alberto Savoia 4
  • Ben Yu 4
  • Erik Kuefler 4
  • Philip Zembrod 4
  • Shyam Seshadri 4
  • Chrome 3
  • Dillon Bly 3
  • John Thomas 3
  • Lesley Katzen 3
  • Marc Kaplan 3
  • Markus Clermont 3
  • Max Kanat-Alexander 3
  • Sonal Shah 3
  • APIs 2
  • Abhishek Arya 2
  • Alan Myrvold 2
  • Alek Icev 2
  • Android 2
  • April Fools 2
  • Chaitali Narla 2
  • Chris Lewis 2
  • Chrome OS 2
  • Diego Salas 2
  • Dori Reuveni 2
  • Jason Arbon 2
  • Jochen Wuttke 2
  • Kostya Serebryany 2
  • Marc Eaddy 2
  • Marko Ivanković 2
  • Mobile 2
  • Oliver Chang 2
  • Simon Stewart 2
  • Stefan Kennedy 2
  • Test Flakiness 2
  • Titus Winters 2
  • Tony Voellm 2
  • WebRTC 2
  • Yiming Sun 2
  • Yvette Nameth 2
  • Zuri Kemp 2
  • Aaron Jacobs 1
  • Adam Porter 1
  • Adam Raider 1
  • Adel Saoud 1
  • Alan Faulkner 1
  • Alex Eagle 1
  • Amy Fu 1
  • Anantha Keesara 1
  • Antoine Picard 1
  • App Engine 1
  • Ari Shamash 1
  • Arif Sukoco 1
  • Benjamin Pick 1
  • Bob Nystrom 1
  • Bruce Leban 1
  • Carlos Arguelles 1
  • Carlos Israel Ortiz García 1
  • Cathal Weakliam 1
  • Christopher Semturs 1
  • Clay Murphy 1
  • Dagang Wei 1
  • Dan Maksimovich 1
  • Dan Shi 1
  • Dan Willemsen 1
  • Dave Chen 1
  • Dave Gladfelter 1
  • David Bendory 1
  • David Mandelberg 1
  • Derek Snyder 1
  • Diego Cavalcanti 1
  • Dmitry Vyukov 1
  • Eduardo Bravo Ortiz 1
  • Ekaterina Kamenskaya 1
  • Elliott Karpilovsky 1
  • Elliotte Rusty Harold 1
  • Espresso 1
  • Felipe Sodré 1
  • Francois Aube 1
  • Gene Volovich 1
  • Google+ 1
  • Goran Petrovic 1
  • Goranka Bjedov 1
  • Hank Duan 1
  • Havard Rast Blok 1
  • Hongfei Ding 1
  • Jason Elbaum 1
  • Jason Huggins 1
  • Jay Han 1
  • Jeff Hoy 1
  • Jeff Listfield 1
  • Jessica Tomechak 1
  • Jim Reardon 1
  • Joe Allan Muharsky 1
  • Joel Hynoski 1
  • John Micco 1
  • John Penix 1
  • Jonathan Rockway 1
  • Jonathan Velasquez 1
  • Josh Armour 1
  • Julie Ralph 1
  • Kai Kent 1
  • Kanu Tewary 1
  • Karin Lundberg 1
  • Kaue Silveira 1
  • Kevin Bourrillion 1
  • Kevin Graney 1
  • Kirkland 1
  • Kurt Alfred Kluever 1
  • Manjusha Parvathaneni 1
  • Marek Kiszkis 1
  • Marius Latinis 1
  • Mark Ivey 1
  • Mark Manley 1
  • Mark Striebeck 1
  • Matt Lowrie 1
  • Meredith Whittaker 1
  • Michael Bachman 1
  • Michael Klepikov 1
  • Mike Aizatsky 1
  • Mike Wacker 1
  • Mona El Mahdy 1
  • Noel Yap 1
  • Palak Bansal 1
  • Patricia Legaspi 1
  • Per Jacobsson 1
  • Peter Arrenbrecht 1
  • Peter Spragins 1
  • Phil Norman 1
  • Phil Rollet 1
  • Pooja Gupta 1
  • Project Showcase 1
  • Radoslav Vasilev 1
  • Rajat Dewan 1
  • Rajat Jain 1
  • Rich Martin 1
  • Richard Bustamante 1
  • Roshan Sembacuttiaratchy 1
  • Ruslan Khamitov 1
  • Sam Lee 1
  • Sean Jordan 1
  • Sebastian Dörner 1
  • Sharon Zhou 1
  • Shiva Garg 1
  • Siddartha Janga 1
  • Simran Basi 1
  • Stan Chan 1
  • Stephen Ng 1
  • Tejas Shah 1
  • Test Analytics 1
  • Test Engineer 1
  • Tim Lyakhovetskiy 1
  • Tom O'Neill 1
  • Vojta Jína 1
  • automation 1
  • dead code 1
  • iOS 1
  • mutation testing 1


Archive


  • ►  2025 (1)
    • ►  Jan (1)
  • ►  2024 (13)
    • ►  Dec (1)
    • ►  Oct (1)
    • ►  Sep (1)
    • ►  Aug (1)
    • ►  Jul (1)
    • ►  May (3)
    • ►  Apr (3)
    • ►  Mar (1)
    • ►  Feb (1)
  • ►  2023 (14)
    • ►  Dec (2)
    • ►  Nov (2)
    • ►  Oct (5)
    • ►  Sep (3)
    • ►  Aug (1)
    • ►  Apr (1)
  • ►  2022 (2)
    • ►  Feb (2)
  • ►  2021 (3)
    • ►  Jun (1)
    • ►  Apr (1)
    • ►  Mar (1)
  • ►  2020 (8)
    • ►  Dec (2)
    • ►  Nov (1)
    • ►  Oct (1)
    • ►  Aug (2)
    • ►  Jul (1)
    • ►  May (1)
  • ►  2019 (4)
    • ►  Dec (1)
    • ►  Nov (1)
    • ►  Jul (1)
    • ►  Jan (1)
  • ►  2018 (7)
    • ►  Nov (1)
    • ►  Sep (1)
    • ►  Jul (1)
    • ►  Jun (2)
    • ►  May (1)
    • ►  Feb (1)
  • ►  2017 (17)
    • ►  Dec (1)
    • ►  Nov (1)
    • ►  Oct (1)
    • ►  Sep (1)
    • ►  Aug (1)
    • ►  Jul (2)
    • ►  Jun (2)
    • ►  May (3)
    • ►  Apr (2)
    • ►  Feb (1)
    • ►  Jan (2)
  • ►  2016 (15)
    • ►  Dec (1)
    • ►  Nov (2)
    • ►  Oct (1)
    • ►  Sep (2)
    • ►  Aug (1)
    • ►  Jun (2)
    • ►  May (3)
    • ►  Apr (1)
    • ►  Mar (1)
    • ►  Feb (1)
  • ►  2015 (14)
    • ►  Dec (1)
    • ►  Nov (1)
    • ►  Oct (2)
    • ►  Aug (1)
    • ►  Jun (1)
    • ►  May (2)
    • ►  Apr (2)
    • ►  Mar (1)
    • ►  Feb (1)
    • ►  Jan (2)
  • ►  2014 (24)
    • ►  Dec (2)
    • ►  Nov (1)
    • ►  Oct (2)
    • ►  Sep (2)
    • ►  Aug (2)
    • ►  Jul (3)
    • ►  Jun (3)
    • ►  May (2)
    • ►  Apr (2)
    • ►  Mar (2)
    • ►  Feb (1)
    • ►  Jan (2)
  • ▼  2013 (16)
    • ►  Dec (1)
    • ▼  Nov (1)
      • WebRTC Audio Quality Testing
    • ►  Oct (1)
    • ►  Aug (2)
    • ►  Jul (1)
    • ►  Jun (2)
    • ►  May (2)
    • ►  Apr (2)
    • ►  Mar (2)
    • ►  Jan (2)
  • ►  2012 (11)
    • ►  Dec (1)
    • ►  Nov (2)
    • ►  Oct (3)
    • ►  Sep (1)
    • ►  Aug (4)
  • ►  2011 (39)
    • ►  Nov (2)
    • ►  Oct (5)
    • ►  Sep (2)
    • ►  Aug (4)
    • ►  Jul (2)
    • ►  Jun (5)
    • ►  May (4)
    • ►  Apr (3)
    • ►  Mar (4)
    • ►  Feb (5)
    • ►  Jan (3)
  • ►  2010 (37)
    • ►  Dec (3)
    • ►  Nov (3)
    • ►  Oct (4)
    • ►  Sep (8)
    • ►  Aug (3)
    • ►  Jul (3)
    • ►  Jun (2)
    • ►  May (2)
    • ►  Apr (3)
    • ►  Mar (3)
    • ►  Feb (2)
    • ►  Jan (1)
  • ►  2009 (54)
    • ►  Dec (3)
    • ►  Nov (2)
    • ►  Oct (3)
    • ►  Sep (5)
    • ►  Aug (4)
    • ►  Jul (15)
    • ►  Jun (8)
    • ►  May (3)
    • ►  Apr (2)
    • ►  Feb (5)
    • ►  Jan (4)
  • ►  2008 (75)
    • ►  Dec (6)
    • ►  Nov (8)
    • ►  Oct (9)
    • ►  Sep (8)
    • ►  Aug (9)
    • ►  Jul (9)
    • ►  Jun (6)
    • ►  May (6)
    • ►  Apr (4)
    • ►  Mar (4)
    • ►  Feb (4)
    • ►  Jan (2)
  • ►  2007 (41)
    • ►  Oct (6)
    • ►  Sep (5)
    • ►  Aug (3)
    • ►  Jul (2)
    • ►  Jun (2)
    • ►  May (2)
    • ►  Apr (7)
    • ►  Mar (5)
    • ►  Feb (5)
    • ►  Jan (4)

Feed

  • Google
  • Privacy
  • Terms