So there are a couple of different approaches to mocking. It looks like you’ve researched at least one annotation that helps you mock in Python. Sweet! Before we do that, there are some more fundamental steps we can take to help make testing easier, and to replace the functionality we don’t want to test when we’re in a test method.

You want to test this method, whose function (I’m inferring from reading the code) is to

1. iterate over a list of things,
2. map them to some other things,
3. create a file
4. write the other things into the file
5. put that file somewhere via sftp
6. delete the things from the database.

Does that sound correct?

But you want to test this without performing steps 3, 5, or 6.

So, by deduction, I’m getting that you do want to test steps 1, 2, and 4. Is that correct?

I’d also argue for testing step 3. You might want to make sure that file is created, has the data, and has the right filename (especially if some crawler is going to pick up your file at the other end of the drop location based on its name). But maybe you don’t want to because it’s too big or for some other reasons that I don’t know. Either way, the process I’m about to recommend is largely the same.

So, let’s separate your steps into separate methods. Perhaps you have:

a) a method that does steps 1 and 2
b) a method that does step 3 (and maybe 4)
c) a method that does step 5
d) a method that does step 6

Now we have separated the logic we want to perform in tests and the logic we do not want to perform in tests. Sweet.

import pysftp

cnopts = pysftp.CnOpts()
cnopts.hostkeys = None

def do_things(ids, 
              step1=self.map_stuff_from, 
              step2=self.create_and_write_file_with,
              step3=self.sftp_transfer,
              step4=self.delete_stuff_from
              ):
    stuff_from_ids = step1.call(ids)
    file = step2.call(stuff_from_ids)

    success, error = step3.call(file)
    step4.call(ids) if success else logging.log('Error connecting', exception=error)

def map_stuff_from(ids):
    for id in ids:
        things = []
        for foo in thing.checkid(id):
            things.extend(#getsomestuff)
    return things

def create_and_write_file_with(stuff_from_ids):
    #create and write file
    #return file

def sftp_transfer(file):
    try:
        with pysftp.Connection(host, username=username, password=password, cnopts=cnopts) as sftp:
            sftp.put(csv_path)
            sftp.close()
            # delete some values from the db
        except Exception as e:
            return False, e.message

    return true, '' #empty string instead of None so that clients can shove this value into a string without checking for None

def delete_stuff_from(ids):
    #do the deleting

So now our method do_stuff accepts some additional parameters. We’re passing in functions (yay for functions as first class objects!) that do the steps we want to do. But we don’t want to perform all of these functions as-is in our tests. So in our tests, we can make alternative functions for those steps we don’t want to perform and pass them in during testing.

def test_do_things():
    test_ids = [12324,353453,34242]
    
    do_things(test_ids,
       step1=map_stuff_from_ids,
       step2=create_and_write_file_with,
       step3=standin_stfp_method_that_does_nothing,
       step4=standin_delete_method_that_does_nothing)
    
    #assert some stuff in the test
    
def standin_sftp_method_that_does_nothing(file):
    return True, ''

def standin_delete_method_that_does_nothing(ids):
    pass #For evaluation purposes only

or, if you’d like more granular assertions, you can test each of your step methods individually:

def test_map_stuff_from_ids():
    test_ids = [12324,353453,34242]

    assert map_stuff_from_ids(test_ids)
       == [Object1, Object2, Object3]

def test_create_and_write_file():
    stuff_from_ids = [Object1, Object2, Object3]
    file = create_and_write_file_with(stuff_from_ids)
    
    assert file.name == 'AwesomeFileName.json'
    
#...and so on

Separating the logic you want to test from the logic you don’t want to test gives you the opportunity to introduce stand-ins in only those places where you need to avoid side-effects, while executing and asserting on everything else.

In this particular case, it also gives your code some modularity and legibility that’s harder to achieve with a large method that does many things.

Chelsea, why did you treat functions as first class objects instead of creating classes?

The code in the question does not appear to live inside an object. In python, ruby, and other scripting languages, you can do this.

I have design justifications for putting code in classes instead, but that wasn’t the question. I wanted to answer the question without introducing the additional component of class-based software design. That said, we are now in a blog post and not the question, so we will go a step further and discuss that architecture now :).

An object-oriented approach would identify separate responsibilities in this code and create a class to handle each of those responsibilities. For example, modifying files, transferring files, and making changes to the database all sound to me like separate responsibilities, each communicating with separate collaborators (the files, an sftp drop point, and the database, respectively).

the methods that contained behavior meant to be tested and the methods that contained behavior not to be tested into two different classes, with a third class that combined them.

So it might look something like this:

import pysftp

class FileProcessor():

     def __init__(self, file_manager, sftp_manager, database_manager): 
         self.file_manager = file_manager
         self.sftp_manager = sftp_manager
         self.database_manager = database_manager

     def do_things(self, ids): 
         stuff_from_ids = self.map_stuff_from(ids)
         file = file_manager.create_and_write_file_with(stuff_from_ids)

         success, error = sftp_manager.sftp_transfer(file)
         database_manager.delete_stuff_from(ids) if success else logging.log('Error connecting', exception=error)

     def map_stuff_from(self, ids):
         for id in ids:
             things = []
             for foo in thing.checkid(id):
                 things.extend(#getsomestuff)
         return things

class FileManager():

     def create_and_write_file_with(self, sstuff_from_ids):
         #create and write file
         #return file

class SftpManager():

     def __init__(self): 
         self.cnopts = pysftp.CnOpts()
         self.cnopts.hostkeys = None

     def sftp_transfer(self, file):
         try:
             with pysftp.Connection(host, username=username, password=password, cnopts=cnopts) as sftp:
                 sftp.put(csv_path)
                 sftp.close()
                 # delete some values from the db
             except Exception as e:
                 return False, e.message

         return true, '' #empty string instead of None so that clients can shove this value into a string without checking for None
 
class DatabaseManager():

     def delete_stuff_from(self, ids):
         #do the deleting

I am not intimately familiar enough with the functionality we’re talking about to definitively say that these are the correct class delineations for this code. However, this exemplifies dividing methods into classes so that each class has one responsibility. For each of these, then, a test double could be rolled like this:

class DummyDatabaseManager(DatabaseManager):

     def delete_stuff_from(self, ids):
         pass

and then injected at test time like so:

def test_do_things()
  dummy_database_manager = DummyDatabaseManager() 
  dummy_sftp_manager = DummySftpManager()
  dummy_file_manager = DummyFileManager()
  subject_under_test = FileProcessor(dummy_file_manager, dummy_sftp_manager, dummy_database_manager)
  subject_under_test.do_things([1,2,3])
  # assert some stuff

def _transfer_file(self, my_file):
            try:
                    with pysftp.Connection('host', username=username, password=password) as sftp:
                    sftp.put(my_file)
                    sftp.close()
                    return True

            except Exception as e:
                    # log an error with exception
                    return None