Testing Blog

Testing on the Toilet: Don’t Overuse Mocks

Tuesday, May 28, 2013
Share on Twitter Share on Facebook
Google
Labels: Andrew Trenk , TotT

21 comments :

  1. Will DalyMay 28, 2013 at 4:24:00 PM PDT

    "Sometimes you can't use a real dependency in a test (e.g. if it's too slow or talks over the network), but there may better options than using mocks"

    Often the easiest solution is to refactor interfaces. In this case, the PaymentProcessor probably shouldn't be searching so deep into the CreditCardServer's dependencies. One possibility is to create a TransactionCreditCardServer that wraps communication with the server in transactions. Then the software under test can ask the TransactionCreditCardServer for the payment directly, making the mocks much easier to set up.

    ReplyDelete
    Replies
    1. AndrewMay 29, 2013 at 9:15:00 AM PDT

      Refactoring might help reduce the problem, but if you use mocks to test TransactionCreditCardServer then you still have the same problem. Plus you'll probably be adding complexity if you add lots of new classes just to make your mocks happy.

      Also, in many cases you won't be able to refactor the mocks away. In the above example there's only one dependency, but what if your code takes five dependencies? The mocking problem gets much worse, and it probably won't be easy to refactor all those away behind a simple interface.

      Delete
      Replies
        Reply
    2. Stanislav BashkyrtsevJuly 18, 2017 at 1:53:00 AM PDT

      I haven't used mocking frameworks quite for some time and I think the formula for this is:
      1. Using Rich Model - moving parts of the logic into the entities instead of keeping them into the stateless services. This way you can test the business logic without dependencies.
      2. Writing Component (a.k.a. Integration) tests for the rest of the tests. Because most tests will be covering business logic, what's left is to test the collaboration between Services, DAOs, etc. So there will be fewer of these tests than usually.

      Check out examples here: http://qala.io/blog/anaemic-architecture-enemy-of-testing.html

      Delete
      Replies
        Reply
    3. Reply
  2. Eric JainMay 28, 2013 at 4:48:00 PM PDT

    Much of the ugliness in this example is due to the mock object being a "partial mock", see http://monkeyisland.pl/2009/01/13/subclass-and-override-vs-partial-mocking-vs-refactoring/ for a discussion when to do partial mocks vs subclass-and-override.

    ReplyDelete
    Replies
      Reply
  3. robannMay 29, 2013 at 1:01:00 AM PDT

    Brilliant and concise article.

    The problem with mocks is they duplicate the logic of the code itself - instantly breaking DRY. Having to use mocks is a code smell and normally indicates the need to refactor.

    ReplyDelete
    Replies
    1. AndrewMay 29, 2013 at 2:00:00 PM PDT

      Mocks work better if you don't put logic into them and just use them to get your code into a certain state (e.g. if you have special handing in your code when your dependency has an empty list, you can use a mock to tell the dependency to return an empty list so your test can execute that code).

      Mocks also work well for testing interactions
      (see http://googletesting.blogspot.com/2013/03/testing-on-toilet-testing-state-vs.html).

      Delete
      Replies
        Reply
    2. CharlesJuly 30, 2013 at 7:02:00 AM PDT

      I don't read any general "problem with mocks" into the article above at all. I have seen tests like the bad example above; those are simply attempts to write integration tests in a unit test style. However mocking is a very useful technique for truly testing single units in isolation (for instance how one of the 5 subsystems represented in the above test interacts with a single other subsystem).

      I would say the opposite. The need to use mocks is not a code smell. The *inability to use mocks properly to write readable concise tests* is a code smell. It means your code is too tightly coupled, and the contracts between the individual components are opaque and poorly understood, and only testable by throwing the whole kit-n-caboodle together, and in the event of a failure, hoping you have good enough logging to figure out exactly what went wrong.

      Delete
      Replies
        Reply
    3. uriDecember 30, 2014 at 1:25:00 PM PST

      I'm intrigued @Charles. Recently saw a talk that said a reason why most TDD sucks is because programmers tend to use mocks to test the implementation and not behavior (and so produce tests with high coupling with the code). More or less, the speaker was saying the same thing as you.

      Any pointers to well-written tests that use mocks?

      Delete
      Replies
        Reply
    4. rliesenfeldOctober 19, 2016 at 12:29:00 PM PDT

      After a decade developing (and using) a mocking library, I could only come to the conclusion that there is hardly any such thing as "well-written tests that use mocks". Generally, such tests are not nearly as good as the equivalent integration tests which I write on a daily basis.

      Delete
      Replies
        Reply
    5. Reply
  4. AnonymousMay 29, 2013 at 6:52:00 AM PDT

    Also it's called unit testing for a reason, testing dependencies is a nono. Although I agree mocking shouldn't be overdone, having test spill outside the unit is not good either. It has just as much potential as mocks to introduce unwanted side-effects.

    ReplyDelete
    Replies
    1. AndrewMay 29, 2013 at 9:26:00 AM PDT

      You can't test every class in isolation. So you can do things like refactor your code so that they don't even know about the depenency (e.g. if you have business logic and database logic mixed together, you can move the business logic into a separate class that has no knowledge of the database, which makes it easier to test).

      But at some point you're going to test your code that uses a dependency, and you can't always use mocks just so you can ignore the dependency.

      Delete
      Replies
        Reply
    2. droogansMay 29, 2013 at 7:49:00 PM PDT

      So that's what end to end tests are for. I agree, unit tests, first and foremost, must compile in ~500ms, tops. Otherwise you can't hook them into your save functions of your editor without noticing the lag, and you lose that tight feedback loop that makes unit tests so critical.

      Let end to end tests run for 20 minutes. You're only running them when you push to production, which is probably only three times a day for your team, anyway.

      Delete
      Replies
        Reply
    3. Reply
  5. Lex PattisonMay 30, 2013 at 9:04:00 AM PDT

    Mocks are like hard drugs... the more you use, the more separated from reality everything becomes.

    ReplyDelete
    Replies
      Reply
  6. UnknownJuly 3, 2013 at 9:21:00 AM PDT

    The whole point of unit testing is that you are attempting to test a unit of functionality. A unit is usually understood as a class. However, sometimes the purpose of that class is to interact with files, databases, networks, other classes, etc., and dragging those into your test muddies the water of what a unit is. Mocking the interaction with the rest of the system is a valid use of mocking I believe.

    There is scenario where mocking cannot be avoided; testing the operation of the program as a result of human interaction. I recently mocked a dialog box service such that I could mock that the user clicked 'yes' or 'no' as a result of being shown a dialog. If you intend to do automated testing of human interaction, mocking is your only way.

    ReplyDelete
    Replies
    1. AndrewJuly 9, 2013 at 6:55:00 PM PDT

      A better definition of a "unit" would be a single class or a group of related classes. Otherwise if you do something like refactor some utility code out of a class into a helper class, that helper class becomes a separate unit, meaning what was previously an implementation detail now becomes something you would have to mock out, and this could easily lead to tests that are hard to maintain.

      But mocking can still get complex sometimes even if you're trying to mock something that's a separate unit, and it really only works well if you have a simple interface that's being mocked out (e.g. if the code under test calls a method that returns a list and your code then does something interesting with that list, you can tell the mock to return an empty list, a list with one element, etc). But if there are more complex interactions that require several mocks to talk to each other or require the mocks to return other mocks, it's a much better idea to use a fake, or possibly even the real thing if it's not too heavyweight.

      Delete
      Replies
        Reply
    2. Reply
  7. CharlesJuly 30, 2013 at 6:54:00 AM PDT

    Doesn't the code being tested above involve a pretty clear Law Of Demeter violation? Code that reaches through 4 levels of dependencies (server -> transactionManager -> transaction -> payment) is going to be difficult to test by any means. If you're truly trying to test the integration between all those systems, then we're talking about an integration test, and mocking is obviously not appropriate there. If you're unit testing, the code should either be refactored not to reach through so many layers, or the component interactions should be tested piecemeal as true units.

    ReplyDelete
    Replies
    1. AndrewAugust 4, 2013 at 5:58:00 PM PDT

      There's really no such thing as a "violation" of the Law of Demeter since it's more of a best practice than a law, and it doesn't necessarily make sense to follow it in every case. And in some situations you might not have a choice if the API has already been designed and you're stuck with that design.

      Using mocks when what you really need is an integration test is already mentioned in the post. This is probably one of the most common cases of overuse of mocks, but similar problems can still happen in unit tests if you indiscriminately mock out all dependencies (e.g. mocking out a bunch of helper classes that are closely related to the class under test).

      Delete
      Replies
        Reply
    2. CharlesAugust 9, 2013 at 1:57:00 PM PDT

      Heh, can I lobby that we call it something other than a "law" if it can't be violated? (Joking) Anyway I was making reference to previous Google Testing Blog posts that advocate writing code with the LoD in mind:
      http://googletesting.blogspot.com/2008/07/how-to-write-3v1l-untestable-code.html
      http://googletesting.blogspot.com/2008/07/breaking-law-of-demeter-is-like-looking.html

      It seems like if you control the code in question, you could rewrite it with the above posts in mind to make it more easily testable. If you don't control the APIs as you suggest, I think I'd write a Facade that I could control whose implementation I left only tested as part of a full system test, and was simple enough that I was okay with that trade off, then the code on top of it talked to the Facade API that I could control (and possibly write a fake of, etc.)

      I do agree that overmocking can get ugly fast, especially in the cast of closely related dependencies like you mention. I guess the reason I initially bristled at your post is that I am somewhat of a test practices evangelist at my company, and a significant portion of the culture here is resistant to the use of *any kind of testing double whatsoever under any circumstance*, that I wanted to clarify it so someone wouldn't use this post (erroneously, I hope) in defense of a never-use-doubles-ever philosophy.

      Delete
      Replies
        Reply
    3. Reply
  8. JosuéSeptember 25, 2013 at 6:07:00 AM PDT

    I suggest the book Growing Object Oriented Software Guided by Tests (GOOS) to see a good use of mocks and TDD.

    There are two styles of TDD: state based (the book Test Driven Development by Example from Kent Beck is a good example) and interactive based (the book GOOS is a good example). As the names suggests state based TDD use much more the state of the objects to construct its tests and interactive TDD use much more the interactions (messages) between objects. Both have pros and cons. Have used the two practices, today I tend to use more interactive TDD, but depending on the problem, I also use state based. And I know of very good programmers in both sides.


    For more information two good groups (IMO) are:

    TDD group
    http://groups.yahoo.com/neo/groups/testdrivendevelopment/info

    Growing Object-Oriented Software (the GOOS book group)
    https://groups.google.com/forum/#!forum/growing-object-oriented-software


    ReplyDelete
    Replies
      Reply
  9. Reed G. LawApril 28, 2014 at 7:06:00 AM PDT

    The immediate problem I see with this test is that it actually makes a $5 charge through the payment processor. Do you really want to have to cancel a charge every time you run your test suite? And what if there's some fee involved? I think those considerations would be enough to make me want to run a test suite as rarely as possible, which is exactly what you don't want from your test suite.

    ReplyDelete
    Replies
    1. AndrewApril 28, 2014 at 10:48:00 AM PDT

      You definitely don't want your tests to talk to a real credit card server. The last paragraph of the post mentions what to do here: either start up a local version of the credit card server for the test (although this might not be feasible if the server is too big and complex), or use a fake in-memory version of the credit card server.

      That's the problem with mocks: they make it easy to write tests for your code, but it's also easy to use them improperly, which can make the tests useless and will likely slow down development too.

      Delete
      Replies
        Reply
    2. Reply
Add comment
Load more...

The comments you read and contribute here belong only to the person who posted them. We reserve the right to remove off-topic comments.

  

Labels


  • TotT 104
  • GTAC 61
  • James Whittaker 42
  • Misko Hevery 32
  • Code Health 31
  • Anthony Vallone 27
  • Patrick Copeland 23
  • Jobs 18
  • Andrew Trenk 13
  • C++ 11
  • Patrik Höglund 8
  • JavaScript 7
  • Allen Hutchison 6
  • George Pirocanac 6
  • Zhanyong Wan 6
  • Harry Robinson 5
  • Java 5
  • Julian Harty 5
  • Adam Bender 4
  • Alberto Savoia 4
  • Ben Yu 4
  • Erik Kuefler 4
  • Philip Zembrod 4
  • Shyam Seshadri 4
  • Chrome 3
  • Dillon Bly 3
  • John Thomas 3
  • Lesley Katzen 3
  • Marc Kaplan 3
  • Markus Clermont 3
  • Max Kanat-Alexander 3
  • Sonal Shah 3
  • APIs 2
  • Abhishek Arya 2
  • Alan Myrvold 2
  • Alek Icev 2
  • Android 2
  • April Fools 2
  • Chaitali Narla 2
  • Chris Lewis 2
  • Chrome OS 2
  • Diego Salas 2
  • Dori Reuveni 2
  • Jason Arbon 2
  • Jochen Wuttke 2
  • Kostya Serebryany 2
  • Marc Eaddy 2
  • Marko Ivanković 2
  • Mobile 2
  • Oliver Chang 2
  • Simon Stewart 2
  • Stefan Kennedy 2
  • Test Flakiness 2
  • Titus Winters 2
  • Tony Voellm 2
  • WebRTC 2
  • Yiming Sun 2
  • Yvette Nameth 2
  • Zuri Kemp 2
  • Aaron Jacobs 1
  • Adam Porter 1
  • Adam Raider 1
  • Adel Saoud 1
  • Alan Faulkner 1
  • Alex Eagle 1
  • Amy Fu 1
  • Anantha Keesara 1
  • Antoine Picard 1
  • App Engine 1
  • Ari Shamash 1
  • Arif Sukoco 1
  • Benjamin Pick 1
  • Bob Nystrom 1
  • Bruce Leban 1
  • Carlos Arguelles 1
  • Carlos Israel Ortiz García 1
  • Cathal Weakliam 1
  • Christopher Semturs 1
  • Clay Murphy 1
  • Dagang Wei 1
  • Dan Maksimovich 1
  • Dan Shi 1
  • Dan Willemsen 1
  • Dave Chen 1
  • Dave Gladfelter 1
  • David Bendory 1
  • David Mandelberg 1
  • Derek Snyder 1
  • Diego Cavalcanti 1
  • Dmitry Vyukov 1
  • Eduardo Bravo Ortiz 1
  • Ekaterina Kamenskaya 1
  • Elliott Karpilovsky 1
  • Elliotte Rusty Harold 1
  • Espresso 1
  • Felipe Sodré 1
  • Francois Aube 1
  • Gene Volovich 1
  • Google+ 1
  • Goran Petrovic 1
  • Goranka Bjedov 1
  • Hank Duan 1
  • Havard Rast Blok 1
  • Hongfei Ding 1
  • Jason Elbaum 1
  • Jason Huggins 1
  • Jay Han 1
  • Jeff Hoy 1
  • Jeff Listfield 1
  • Jessica Tomechak 1
  • Jim Reardon 1
  • Joe Allan Muharsky 1
  • Joel Hynoski 1
  • John Micco 1
  • John Penix 1
  • Jonathan Rockway 1
  • Jonathan Velasquez 1
  • Josh Armour 1
  • Julie Ralph 1
  • Kai Kent 1
  • Kanu Tewary 1
  • Karin Lundberg 1
  • Kaue Silveira 1
  • Kevin Bourrillion 1
  • Kevin Graney 1
  • Kirkland 1
  • Kurt Alfred Kluever 1
  • Manjusha Parvathaneni 1
  • Marek Kiszkis 1
  • Marius Latinis 1
  • Mark Ivey 1
  • Mark Manley 1
  • Mark Striebeck 1
  • Matt Lowrie 1
  • Meredith Whittaker 1
  • Michael Bachman 1
  • Michael Klepikov 1
  • Mike Aizatsky 1
  • Mike Wacker 1
  • Mona El Mahdy 1
  • Noel Yap 1
  • Palak Bansal 1
  • Patricia Legaspi 1
  • Per Jacobsson 1
  • Peter Arrenbrecht 1
  • Peter Spragins 1
  • Phil Norman 1
  • Phil Rollet 1
  • Pooja Gupta 1
  • Project Showcase 1
  • Radoslav Vasilev 1
  • Rajat Dewan 1
  • Rajat Jain 1
  • Rich Martin 1
  • Richard Bustamante 1
  • Roshan Sembacuttiaratchy 1
  • Ruslan Khamitov 1
  • Sam Lee 1
  • Sean Jordan 1
  • Sebastian Dörner 1
  • Sharon Zhou 1
  • Shiva Garg 1
  • Siddartha Janga 1
  • Simran Basi 1
  • Stan Chan 1
  • Stephen Ng 1
  • Tejas Shah 1
  • Test Analytics 1
  • Test Engineer 1
  • Tim Lyakhovetskiy 1
  • Tom O'Neill 1
  • Vojta Jína 1
  • automation 1
  • dead code 1
  • iOS 1
  • mutation testing 1


Archive


  • ►  2025 (1)
    • ►  Jan (1)
  • ►  2024 (13)
    • ►  Dec (1)
    • ►  Oct (1)
    • ►  Sep (1)
    • ►  Aug (1)
    • ►  Jul (1)
    • ►  May (3)
    • ►  Apr (3)
    • ►  Mar (1)
    • ►  Feb (1)
  • ►  2023 (14)
    • ►  Dec (2)
    • ►  Nov (2)
    • ►  Oct (5)
    • ►  Sep (3)
    • ►  Aug (1)
    • ►  Apr (1)
  • ►  2022 (2)
    • ►  Feb (2)
  • ►  2021 (3)
    • ►  Jun (1)
    • ►  Apr (1)
    • ►  Mar (1)
  • ►  2020 (8)
    • ►  Dec (2)
    • ►  Nov (1)
    • ►  Oct (1)
    • ►  Aug (2)
    • ►  Jul (1)
    • ►  May (1)
  • ►  2019 (4)
    • ►  Dec (1)
    • ►  Nov (1)
    • ►  Jul (1)
    • ►  Jan (1)
  • ►  2018 (7)
    • ►  Nov (1)
    • ►  Sep (1)
    • ►  Jul (1)
    • ►  Jun (2)
    • ►  May (1)
    • ►  Feb (1)
  • ►  2017 (17)
    • ►  Dec (1)
    • ►  Nov (1)
    • ►  Oct (1)
    • ►  Sep (1)
    • ►  Aug (1)
    • ►  Jul (2)
    • ►  Jun (2)
    • ►  May (3)
    • ►  Apr (2)
    • ►  Feb (1)
    • ►  Jan (2)
  • ►  2016 (15)
    • ►  Dec (1)
    • ►  Nov (2)
    • ►  Oct (1)
    • ►  Sep (2)
    • ►  Aug (1)
    • ►  Jun (2)
    • ►  May (3)
    • ►  Apr (1)
    • ►  Mar (1)
    • ►  Feb (1)
  • ►  2015 (14)
    • ►  Dec (1)
    • ►  Nov (1)
    • ►  Oct (2)
    • ►  Aug (1)
    • ►  Jun (1)
    • ►  May (2)
    • ►  Apr (2)
    • ►  Mar (1)
    • ►  Feb (1)
    • ►  Jan (2)
  • ►  2014 (24)
    • ►  Dec (2)
    • ►  Nov (1)
    • ►  Oct (2)
    • ►  Sep (2)
    • ►  Aug (2)
    • ►  Jul (3)
    • ►  Jun (3)
    • ►  May (2)
    • ►  Apr (2)
    • ►  Mar (2)
    • ►  Feb (1)
    • ►  Jan (2)
  • ▼  2013 (16)
    • ►  Dec (1)
    • ►  Nov (1)
    • ►  Oct (1)
    • ►  Aug (2)
    • ►  Jul (1)
    • ►  Jun (2)
    • ▼  May (2)
      • Testing on the Toilet: Don’t Overuse Mocks
      • GTAC 2013 Wrap-up
    • ►  Apr (2)
    • ►  Mar (2)
    • ►  Jan (2)
  • ►  2012 (11)
    • ►  Dec (1)
    • ►  Nov (2)
    • ►  Oct (3)
    • ►  Sep (1)
    • ►  Aug (4)
  • ►  2011 (39)
    • ►  Nov (2)
    • ►  Oct (5)
    • ►  Sep (2)
    • ►  Aug (4)
    • ►  Jul (2)
    • ►  Jun (5)
    • ►  May (4)
    • ►  Apr (3)
    • ►  Mar (4)
    • ►  Feb (5)
    • ►  Jan (3)
  • ►  2010 (37)
    • ►  Dec (3)
    • ►  Nov (3)
    • ►  Oct (4)
    • ►  Sep (8)
    • ►  Aug (3)
    • ►  Jul (3)
    • ►  Jun (2)
    • ►  May (2)
    • ►  Apr (3)
    • ►  Mar (3)
    • ►  Feb (2)
    • ►  Jan (1)
  • ►  2009 (54)
    • ►  Dec (3)
    • ►  Nov (2)
    • ►  Oct (3)
    • ►  Sep (5)
    • ►  Aug (4)
    • ►  Jul (15)
    • ►  Jun (8)
    • ►  May (3)
    • ►  Apr (2)
    • ►  Feb (5)
    • ►  Jan (4)
  • ►  2008 (75)
    • ►  Dec (6)
    • ►  Nov (8)
    • ►  Oct (9)
    • ►  Sep (8)
    • ►  Aug (9)
    • ►  Jul (9)
    • ►  Jun (6)
    • ►  May (6)
    • ►  Apr (4)
    • ►  Mar (4)
    • ►  Feb (4)
    • ►  Jan (2)
  • ►  2007 (41)
    • ►  Oct (6)
    • ►  Sep (5)
    • ►  Aug (3)
    • ►  Jul (2)
    • ►  Jun (2)
    • ►  May (2)
    • ►  Apr (7)
    • ►  Mar (5)
    • ►  Feb (5)
    • ►  Jan (4)

Feed

  • Google
  • Privacy
  • Terms