lambda, map, filter and sorted with key in Python

The lambda is a small function, with no name, written on a single line: lambda n: n * 2 receives n and returns the double. It serves mainly to be passed as an argument to another function, and the three that receive it most are map, which transforms each item of a group, filter, which keeps only the items that pass a test, and sorted, which orders by the value that the key function calculates for each item.

What makes this possible is an idea that appeared in passing in the article on def: in Python, a function is a value, like a number or a text. It can be stored in a variable, passed as an argument and returned from another function. This article closes the functions module by showing that idea in action. We will write lambdas, use map, filter and sorted, see the errors that show up along the way and learn about a behavior of sorted in MicroPython that differs from Python on the computer and can surprise whoever sorts sensor data.

Functions are values

Until now, every function was called by its name, with parentheses. But the name without parentheses, like dobro, is just a reference to the function, and it can be handed to another function:

def aplicar(f, x):
    return f(x)

def dobro(n):
    return n * 2

print(aplicar(dobro, 4))
print(aplicar(lambda n: n + 1, 4))
print(aplicar(abs, -3))
print(aplicar(str, 42) + "!")
f = dobro
print(f(10))
print(type(f))

The output is 8, 5, 3, 42!, 20 and <class 'function'>. The function aplicar does not know what f does, only that it receives a value and hands back another. In the first call, f is dobro. In the second, it is a lambda written on the spot. In the third, it is the ready-made function abs, and in the fourth it is str, which converts to text. The line f = dobro shows that the function can be stored in a variable and called by another name, and type(f) confirms that the value is a function.

This pattern, a function that receives another, appears all the time in Python and is the basis of map, filter and sorted.

lambda: a function of one expression

The lambda is the short way of writing a simple function. The figure shows the same function dobro written both ways. In the lambda, the word replaces def, the parameters come without parentheses, the colon comes after them and what follows is an expression, whose value is returned automatically, without writing return.

def dobro(n):
    return n * 2

dobro_l = lambda n: n * 2
print(dobro(4), dobro_l(4))
print((lambda a, b: a + b)(2, 3))
ola = lambda: "hi"
print(ola())
par = lambda n: "even" if n % 2 == 0 else "odd"
print(par(4), par(7))
pot = lambda base, exp=2: base ** exp
print(pot(5), pot(2, 10))

The output is 8 8, 5, hi, even odd and 25 1024. The lambda accepts zero, one or more parameters, and even default values, like the exp=2 of the power. The line (lambda a, b: a + b)(2, 3) creates the function and calls it at the same point, which is rarely done outside examples. The par shows the conditional expression, value1 if condition else value2, which is the way of putting a decision inside a single expression.

The restriction that defines the lambda is that the body is a single expression. A return, assignments, a multi-line if and loops do not fit:

f = lambda n: return n

Traceback (most recent call last):
  File "main.py", line 1
SyntaxError: invalid syntax

The same goes for an assignment inside the lambda, which gives SyntaxError: can't assign to expression. A print inside a lambda is allowed, because it is a call, and therefore an expression.

The lambda has no name: with print(dobro) it shows <function <lambda> at 0x3ffb0010>, and the address varies. This makes error messages less clear. That is why the rule of good use is the following: if the function will be reused or has more than one line of logic, use def and give it a name. The lambda is for the short piece that will be handed to another function and used right there.

map: transforming each item

map(function, group) applies the function to each item of the group and produces the results. It returns an object that generates the values on demand, and to see the results as a list you have to convert it with list():

numeros = [1, 2, 3, 4]
dobros = map(lambda n: n * 2, numeros)
print(list(dobros))
print(list(map(str, numeros)))
print(list(map(lambda a, b: a + b, [1, 2, 3], [10, 20, 30])))
for d in map(lambda n: n * n, range(1, 5)):
    print(d, end=" ")
print()
print(sum(map(lambda n: n * 2, numeros)))

The output is [2, 4, 6, 8], ['1', '2', '3', '4'], [11, 22, 33], 1 4 9 16 and 20. With two groups, the function receives one item from each at a time, as in the pairwise sum. In a for or in a sum, the map can be used without converting. The number of items in the output is equal to that of the input, as the figure shows.

Two traps of map are worth noting. The first is that the result is not a list:

m = map(lambda n: n * 2, [1, 2, 3])
print(m[0])

Traceback (most recent call last):
  File "main.py", line 2, in <module>
TypeError: 'map' object isn't subscriptable

Python on the computer writes is not subscriptable. The same happens with len(m), which gives object of type 'map' has no len(). The second is that the map is used up after being used once:

m = map(lambda n: n + 1, [1, 2])
print(list(m))
print(list(m))

The output is [2, 3] and then []. The second conversion no longer finds values, because the first consumed everything. When the result will be used more than once, store the list: dobros = list(map(...)).

filter: keeping only what passes

filter(function, group) keeps the items for which the function returns true. The function is a test, like the conditions in the articles on if:

numeros = [1, 2, 3, 4, 5, 6]
pares = filter(lambda n: n % 2 == 0, numeros)
print(list(pares))
print(list(filter(lambda n: n > 3, numeros)))
print(list(filter(None, [0, 1, "", "a", None, 2, [], [3]])))
print(list(filter(lambda s: len(s) > 3, ["pera", "uva", "banana", "kiwi"])))

The output is [2, 4, 6], [4, 5, 6], [1, 'a', 2, [3]] and ['pera', 'banana', 'kiwi']. (The fruit words are kept in Portuguese on purpose: the order the simulator gives to words of the same length depends on them. pera is pear and uva is grape.) The figure shows the first case, with the odd numbers crossed out. The third line uses None in place of the function: the filter then keeps the true items and discards the "empty" ones, such as 0, the empty text, None and the empty list. It is a practical shortcut for cleaning up a group.

The filter has the same on-demand object behavior as the map, and the same traps: it needs list() to become a list and it is used up after being used. Type errors appear when the second argument is not a group:

print(list(filter(lambda n: n > 1, 5)))

Traceback (most recent call last):
  File "main.py", line 1, in <module>
TypeError: 'int' object isn't iterable

Python on the computer says 'int' object is not iterable.

sorted: ordering with key

sorted(group) returns a new list with the items in ascending order, without changing the original. With reverse=True, the order is descending. And with key=function, the ordering stops comparing the items directly and compares the value that the function calculates for each one:

nums = [3, 1, 2]
print(sorted(nums))
print(sorted(nums, reverse=True))
print(nums)
palavras = ["pera", "uva", "banana"]
print(sorted(palavras))
print(sorted(palavras, key=len))
print(sorted(palavras, key=len, reverse=True))

The output is [1, 2, 3], [3, 2, 1], [3, 1, 2], ['banana', 'pera', 'uva'], ['uva', 'pera', 'banana'] and ['banana', 'pera', 'uva']. nums is still [3, 1, 2], because sorted does not touch it. Without key, the texts are sorted in alphabetical order. With key=len, they are sorted by length: uva (3), pera (4), banana (6), and with reverse=True the order is reversed. The figure shows the case of key=len. Here all the keys are different, and that matters for the next point.

The method lista.sort() sorts the list itself, in place, and returns None:

lista = [3, 1, 2]
r = lista.sort()
print(r, lista)
nova = sorted([3, 1, 2])
print(nova)

The output is None [1, 2, 3] and [1, 2, 3]. Writing lista = lista.sort() is a classic mistake: the variable comes to hold None and the list is lost.

The key is very useful with tuples, when each item has several parts:

sensores = [("living", 24.5), ("kitchen", 27.0), ("bedroom", 22.5)]
mais_quente = max(sensores, key=lambda s: s[1])
print("hottest:", mais_quente[0], mais_quente[1])
for nome, temp in sorted(sensores, key=lambda s: s[1]):
    print(nome, temp)
frias = list(filter(lambda s: s[1] < 25, sensores))
print(frias)
nomes = list(map(lambda s: s[0], sensores))
print(nomes)

The output is hottest: kitchen 27.0, the three lines bedroom 22.5, living 24.5 and kitchen 27.0, [('living', 24.5), ('bedroom', 22.5)] and ['living', 'kitchen', 'bedroom']. The s[1] takes the temperature of each tuple. The functions max and min also accept key, and max returns the whole tuple, which makes it possible to know which sensor has the largest value, and not just the value. The map with s[0] extracts only the names.

Attention: MicroPython's sorted is not stable

A sort is stable when items with the same key stay in the same order they were in. Python on the computer guarantees this. MicroPython does not: its code uses a sorting algorithm that can swap the order of tied items, and the project's simulator reproduces the board's algorithm. The same program gives different results:

palavras = ["pera", "uva", "banana", "kiwi"]
print(sorted(palavras, key=len))
print(sorted(palavras, key=lambda s: (len(s), s)))

In the simulator, the output is ['uva', 'kiwi', 'pera', 'banana'] on both lines. On the computer's Python, the first line gives ['uva', 'pera', 'kiwi', 'banana'], with pera before kiwi, as in the original list. The words pera and kiwi have the same length, 4, and tie. The computer keeps the input order. The simulator reverses it.

In practice, when there is a tie, the order among the tied items is not guaranteed on the ESP32. The solution is to eliminate the tie with a composite key, a tuple with the main criterion and a tiebreaker:

leituras = [("a", 30), ("b", 12), ("c", 25), ("d", 12), ("e", 12)]
print(sorted(leituras, key=lambda t: (t[1], t[0])))
print(sorted(["b", "A", "c", "B"], key=lambda s: (s.lower(), s)))

The output is [('b', 12), ('d', 12), ('e', 12), ('c', 25), ('a', 30)] and ['A', 'B', 'b', 'c'], the same in the simulator and on the computer. The key (t[1], t[0]) sorts first by the value and, in ties, by the name, and the result becomes predictable. Whenever the order of the tied items matters, as in a ranking list or in a sensor report, break the tie with a composite key.

Another detail of the simulator, which also reproduces the board: the key function is called at each comparison, and not once per item. Counting the calls on a list of 8 items, the simulator makes 19, and Python on the computer makes 8:

chamadas = 0
def chave(x):
    global chamadas
    chamadas += 1
    return x

lista = [5, 3, 8, 1, 9, 2, 7, 4]
r = sorted(lista, key=chave)
print(r, "key calls:", chamadas, "items:", len(lista))

The output is [1, 2, 3, 4, 5, 7, 8, 9] key calls: 19 items: 8. This matters in two cases: a slow key, which queries a sensor, for example, will be called many times, and a key with a side effect, such as a print, will appear more times than expected. Keep the key short and free of side effects.

Common errors with sorted

Sorting values of incomparable types gives a TypeError:

print(sorted([3, "a", 1]))

Traceback (most recent call last):
  File "main.py", line 1, in <module>
TypeError: unsupported types for __lt__: 'int', 'str'

Python on the computer writes '<' not supported between instances of 'str' and 'int'. The cause is a list that mixes numbers and texts, as happens when a reading arrives as text. Convert first, with int() or float(), or use a key that normalizes. Passing something that is not a group, like sorted(5), gives 'int' object isn't iterable. And a key with the wrong number of parameters also fails:

print(sorted(["a", "b"], key=lambda: 1))

Traceback (most recent call last):
  File "main.py", line 1, in <module>
TypeError: function takes 0 positional arguments but 1 were given

The key always receives one argument, the item, so the lambda needs a parameter, like lambda s: ....

A lambda trap inside loops

When a lambda uses an outside variable, it keeps the name, and not the value from the moment it was created. The value is only read when the lambda is called:

fs = []
for i in range(3):
    fs.append(lambda: i)
print(fs[0](), fs[1](), fs[2]())
gs = []
for i in range(3):
    gs.append(lambda i=i: i)
print(gs[0](), gs[1](), gs[2]())

The output is 2 2 2 and 0 1 2. In the first version, the three lambdas look at the same i, which at the end of the loop is 2. In the second, the i=i is a parameter with a default value, and the default is calculated at creation, so each lambda keeps its own. It is the same mechanism as in the article on scope, and it appears mainly when creating several event handlers in a loop.

On the ESP32

The three features fit well into the handling of readings. Going back to the ADC example, which reads from 0 to 4095, the filter discards the readings outside the range, max and min find the extremes and sorted orders them:

leituras = [512, -1, 530, 4096, 498]
validas = list(filter(lambda v: 0 <= v <= 4095, leituras))
print(validas)
print("largest", max(validas), "smallest", min(validas))
print(sorted(validas, reverse=True))
print("average", sum(validas) / len(validas))
percentual = list(map(lambda v: v * 100 // 4095, validas))
print(percentual)

The output is [512, 530, 498], largest 530 smallest 498, [530, 512, 498], average 513.3333 and [12, 12, 12]. The same set of readings appeared in the article on break and continue, with a loop and a continue; here the filter does the discarding job in one line. The 513.3333 is the simulator's 32-bit float, and Python on the computer would show 513.3333333333334. The article on the ADC explains the reading range.

Lambdas are also the usual way of writing short event handlers. The simulator accepts registering one on a button's interrupt:

from machine import Pin

botao = Pin(4, Pin.IN, Pin.PULL_UP)
botao.irq(handler=lambda p: print("pressed", p), trigger=Pin.IRQ_FALLING)
print("irq configured")

The Serial shows irq configured. The handler is only called when the event happens, and it receives the pin as an argument, the p of the lambda. The handler= and the trigger= are named arguments, as in the article on parameters. If the handler needs more than one line or changes a global variable, write a function with def. The article on programming without freezing deals with how to organize this kind of code.

In the classroom: where these features tend to trip people up

These cases come up again and again in classes.

Forgetting the list() after map or filter

The student prints map(...) and sees <map object at 0x3ffb0020>, thinking that the computer "did not calculate". The object only generates the values when someone asks for them, and list() asks for all of them.

Using the result twice

The first use works and the second comes out empty, because map and filter are used up. The solution is to store the list.

Lambda with more than one line

The student tries to put an if with else on separate lines, or a return, inside the lambda, and gets a SyntaxError. When the logic needs more than one expression, it is time to use def.

Assigning the sorted list to the result of sort

lista = lista.sort() leaves None in the variable. The choice is between lista.sort(), which sorts in place, and lista = sorted(lista), which creates a new one.

Expecting stability in the ESP32's sorted

The program works perfectly on the computer, and on the board the tied items show up in another order. The tiebreaker with a composite key solves it, and it is worth including whenever the order of equal items makes a difference.

Comparing mixed types

A list with 3, "a" and 1. It happens a lot with data read from a file or from an input, where everything arrives as text. Converting with int() before sorting avoids the TypeError.

The loop lambdas that all give the same value

The case of the 2 2 2. The fix is the i=i, and the symptom is always the same: all the functions created in the loop answer with the value of the last step.

Exercise: student ranking

Given a list of students with grades, produce three things: the ranking from highest to lowest grade, the list of those who passed (grade 7 or more) and the list of names in capitals.

alunos = [("Ana", 8.5), ("Bia", 6.0), ("Caio", 9.0), ("Duda", 6.0)]
print(sorted(alunos, key=lambda a: a[1], reverse=True))
aprovados = list(filter(lambda a: a[1] >= 7, alunos))
print(aprovados)
print(list(map(lambda a: a[0].upper(), alunos)))

The output is [('Caio', 9.0), ('Ana', 8.5), ('Bia', 6.0), ('Duda', 6.0)], [('Ana', 8.5), ('Caio', 9.0)] and ['ANA', 'BIA', 'CAIO', 'DUDA']. Notice that Bia and Duda tie with 6.0. In this example their order matched the computer's, but, since MicroPython's sort is not stable, there is no guarantee that this will repeat with other data. To improve it, change the key to lambda a: (-a[1], a[0]), which sorts by descending grade and, in ties, by name, and confirm that the result becomes predictable. The minus sign reverses the grade without needing reverse=True.

Frequently asked questions

What is an anonymous function?

It is a function with no name. The lambda creates one, and that is why Python shows it as <lambda> in messages.

Can I store a lambda in a variable?

You can, as in dobro = lambda n: n * 2. It works, but if the function deserves a name, def is clearer and gives better error messages.

When should I use map and filter, and when a loop?

When the transformation or the test fits in a short expression, both forms work, and the choice is about readability. If the body needs several lines, the loop is better. The next module in the series shows list comprehensions, which do the same job as map and filter in a way that many people find more readable.

Why do both sorted and sort exist?

sorted accepts any group and returns a new list. sort is a list method, sorts the list itself and returns None. Use sorted when you need the original.

What changes in MicroPython?

The main difference is the unstable sort: tied items can come out in another order, and the key is called several times per item. The rest, lambda, map and filter, behaves as on the computer.

Can I use key in min and max?

You can. max(sensores, key=lambda s: s[1]) returns the item with the largest s[1], and not just the value.

Where to go next

This article closes the functions module. If you want to review the basics, go back to the articles on functions, parameters and arguments, return, scope and recursion. The challenges Double Function and Larger of Two can also be solved with lambda, as an optional exercise, and the Logic with Python path gathers all the challenges.

The next article opens the data structures module with lists: how to create, access, change and go through them, which is what several examples in this article have already used without introducing.