Notebook download
Iterators and Generators¶
In Python, anything which can be iterated over is called an iterable:
In[1]:
bowl = { "apple": 5, "banana": 3, "orange": 7}for fruit in bowl: print(fruit.upper())
APPLEBANANAORANGE
Surprisingly often, we want to iterate over something that takes a moderatelylarge amount of memory to store - for example, our map images in thegreen-graph example.
Our green-graph example involved making an array of all the maps between Londonand Birmingham. This kept them all in memory at the same time: first wedownloaded all the maps, then we counted the green pixels in each of them.
This would NOT work if we used more points: eventually, we would run out of memory.We need to use a generator instead. This chapter will look at iterators and generators in more detail:how they work, when to use them, how to create our own.
Iterators¶
Consider the basic python range function:
In[2]:
range(10)
Out[2]:
range(0, 10)
In[3]:
total = 0for x in range(int(1e6)): total += xtotal
Out[3]:
499999500000
In order to avoid allocating a million integers, range actually uses an iterator.
We don't actually need a million integers at once, just eachinteger in turn up to a million.
Because we can get an iterator from it, we say that a range is an iterable.
So we can for-loop over it:
In[4]:
for i in range(3): print(i)
012
There are two important Python built-in functions for working with iterables.First is iter, which lets us create an iterator from any iterable object.
In[5]:
a = iter(range(3))
Once we have an iterator object, we can pass it to the next function. Thismoves the iterator forward, and gives us its next element:
In[6]:
next(a)
Out[6]:
0
In[7]:
next(a)
Out[7]:
1
In[8]:
next(a)
Out[8]:
2
When we are out of elements, a StopIteration exception is raised:
In[9]:
next(a)
---------------------------------------------------------------------------StopIteration Traceback (most recent call last)Cell In [9], line 1----> 1 next(a)StopIteration:
This tells Python that the iteration is over. For example, if we are in a for i in range(3) loop, this lets us know when we should exit the loop.
We can turn an iterable or iterator into a list with the list constructor function:
In[10]:
list(range(5))
Out[10]:
[0, 1, 2, 3, 4]
Defining Our Own Iterable¶
When we write next(a), under the hood Python tries to call the __next__() method of a. Similarly, iter(a) calls a.__iter__().
We can make our own iterators by defining classes that can be used with the next() and iter() functions: this is the iterator protocol.
For each of the concepts in Python, like sequence, container, iterable, the language defines a protocol, a set of methods a class must implement, in order to be treated as a member of that concept.
To define an iterator, the methods that must be supported are __next__() and __iter__().
__next__() must update the iterator.
We'll see why we need to define __iter__ in a moment.
Here is an example of defining a custom iterator class:
In[11]:
class fib_iterator: """An iterator over part of the Fibonacci sequence.""" def __init__(self, limit, seed1=1, seed2=1): self.limit = limit self.previous = seed1 self.current = seed2 def __iter__(self): return self def __next__(self): (self.previous, self.current) = (self.current, self.previous + self.current) self.limit -= 1 if self.limit < 0: raise StopIteration() return self.current
In[12]:
x = fib_iterator(5)
In[13]:
next(x)
Out[13]:
2
In[14]:
next(x)
Out[14]:
3
In[15]:
next(x)
Out[15]:
5
In[16]:
next(x)
Out[16]:
8
In[17]:
for x in fib_iterator(5): print(x)
235813
In[18]:
sum(fib_iterator(1000))
Out[18]:
297924218508143360336882819981631900915673130543819759032778173440536722190488904520034508163846345539055096533885943242814978469042830417586260359446115245634668393210192357419233828310479227982326069668668250
A shortcut to iterables: the __iter__ method¶
In fact, we don't always have to define both __iter__ and __next__!
If, to be iterated over, a class just wants to behave as if it were some other iterable, you can just implement __iter__ and return iter(some_other_iterable), without implementing next. For example, an image class might want to implement some metadata, but behave just as if it were just a 1-d pixel array when being iterated:
In[19]:
from numpy import arrayfrom matplotlib import pyplot as pltclass MyImage(object): def __init__(self, pixels): self.pixels = array(pixels, dtype='uint8') self.channels = self.pixels.shape[2] def __iter__(self): # return an iterator over just the pixel values return iter(self.pixels.reshape(-1, self.channels)) def show(self): plt.imshow(self.pixels, interpolation="None")x = [[[255, 255, 0], [0, 255, 0]], [[0, 0, 255], [255, 255, 255]]]image = MyImage(x)
In[20]:
%matplotlib inlineimage.show()
In[21]:
image.channels
Out[21]:
3
In[22]:
from webcolors import rgb_to_namefor pixel in image: print(rgb_to_name(pixel))
yellowlimebluewhite
See how we used image in a for loop, even though it doesn't satisfy the iterator protocol (we didn't define both __iter__ and __next__ for it)?
The key here is that we can use any iterable object (like image) in a for expression,not just iterators! Internally, Python will create an iterator from the iterable (by calling its __iter__ method), but this means we don't need to define a __next__ method explicitly.
The iterator protocol is to implement both __iter__ and__next__, while the iterable protocol is to implement __iter__ and returnan iterator.
Generators¶
There's a fair amount of "boiler-plate" in the above class-based definition ofan iterable.
Python provides another way to specify somethingwhich meets the iterator protocol: generators.
In[23]:
def my_generator(): yield 5 yield 10x = my_generator()
In[24]:
next(x)
Out[24]:
5
In[25]:
next(x)
Out[25]:
10
In[26]:
next(x)
---------------------------------------------------------------------------StopIteration Traceback (most recent call last)Cell In [26], line 1----> 1 next(x)StopIteration:
In[27]:
for a in my_generator(): print(a)
510
In[28]:
sum(my_generator())
Out[28]:
15
A function which has yield statements instead of a return statement returnstemporarily: it automagically becomes something which implements __next__.
Each call of next() returns control to the function where itleft off.
Control passes back-and-forth between the generator and the caller.Our Fibonacci example therefore becomes a function rather than a class.
In[29]:
def yield_fibs(limit, seed1=1, seed2=1): current = seed1 previous = seed2 while limit > 0: limit -= 1 current, previous = current + previous, current yield current
We can now use the output of the function like a normal iterable:
In[30]:
sum(yield_fibs(5))
Out[30]:
31
In[31]:
for a in yield_fibs(10): if a % 2 == 0: print(a)
2834144
Sometimes we may need to gather all values from a generator into a list, such as before passing them to a function that expects a list:
In[32]:
list(yield_fibs(10))
Out[32]:
[2, 3, 5, 8, 13, 21, 34, 55, 89, 144]
In[33]:
plt.plot(list(yield_fibs(20)))
Out[33]:
[<matplotlib.lines.Line2D at 0x7f5ce1ae6610>]
Iterables and generators can be used to achieve complex behaviour, especially when combined with functional programming. In fact, Python itself contains some very useful language features that make use of these practices: context managers and decorators. We have already seen these in this class, but here we discuss them in more detail.
Context managers¶
We have seen before [notebook] that, instead of separately opening and closeing a file, we can havethe file be automatically closed using a context manager:
In[34]:
%%writefile example.yamlmodelname: brilliant
Writing example.yaml
In[35]:
import yamlwith open('example.yaml') as foo: print(yaml.safe_load(foo))
{'modelname': 'brilliant'}In addition to more convenient syntax, this takes care of any clean-up that has to be done after the file is closed, even if any errors occur while we are working on the file.
How could we define our own one of these, if we too have clean-up code wealways want to run after a calling function has done its work, or set-up codewe want to do first?
We can define a class that meets an appropriate protocol:
In[36]:
class verbose_context(): def __init__(self, name): self.name=name def __enter__(self): print("Get ready, ", self.name) def __exit__(self, exc_type, exc_value, traceback): print("OK, done")with verbose_context("Monty"): print("Doing it!")
Get ready, MontyDoing it!OK, done
However, this is pretty verbose! Again, a generator with yield makes for an easier syntax:
In[37]:
from contextlib import contextmanager@contextmanagerdef verbose_context(name): print("Get ready for action, ", name) yield name.upper() print("You did it")with verbose_context("Monty") as shouty: print(f"Doing it, {shouty}")
Get ready for action, MontyDoing it, MONTYYou did it
Again, we use yield to temporarily return from a function.
Decorators¶
When doing functional programming, we may often want to define mutatorfunctions which take in one function and return a new function, such as ourderivative example earlier.
In[38]:
from math import sqrtdef repeater(count): def wrap_function_in_repeat(func): def _repeated(x): counter = count while counter > 0: counter -= 1 x = func(x) return x return _repeated return wrap_function_in_repeatfiftytimes = repeater(50)fiftyroots = fiftytimes(sqrt)print(fiftyroots(100))
1.000000000000004
It turns out that, quite often, we want to apply one of these to a function as we're defining a class.For example, we may want to specify that after certain methods are called, data should always be stored:
Any function which accepts a function as its first argument and returns a function can be used as a decorator like this.
Much of Python's standard functionality is implemented as decorators: we'veseen @contextmanager, @classmethod and @attribute. The @contextmanagermetafunction, for example, takes in an iterator, and yields a class conformingto the context manager protocol.
In[39]:
@repeater(3)def hello(name): return f"Hello, {name}"
In[40]:
hello("Cleese")
Out[40]:
'Hello, Hello, Hello, Cleese'
Supplementary material¶
The remainder of this page contains an example of the flexibility of the features discussed above. Specifically, it shows how generators and context managers can be combined to create a testing framework like the one previously seen in the course.
Test generators¶
A few weeks ago we saw a test which loaded its test cases from a YAML file andasserted each input with each output. This was nice and concise, but had oneflaw: we had just one test, covering all the fixtures, so we got just one . inthe test output when we ran the tests, and if any test failed, the rest werenot run. We can do a nicer job with a test generator:
In[41]:
def assert_exemplar(**fixture): answer = fixture.pop('answer') assert_equal(greet(**fixture), answer)def test_greeter(): with open(os.path.join(os.path.dirname( __file__), 'fixtures', 'samples.yaml') ) as fixtures_file: fixtures = yaml.safe_load(fixtures_file) for fixture in fixtures: yield assert_exemplar(**fixture)
Each time a function beginning with test_ does a yield it results in another test.
Negative test contexts managers¶
We have seen this:
In[42]:
from pytest import raiseswith raises(AttributeError): x = 2 x.foo()
We can now see how pytest might have implemented this:
In[43]:
from contextlib import contextmanager@contextmanagerdef reimplement_raises(exception): try: yield except exception: pass else: raise Exception("Expected,", exception, " to be raised, nothing was.")
In[44]:
with reimplement_raises(AttributeError): x = 2 x.foo()
Negative test decorators¶
Some frameworks, like nose, also implement a very nice negative test decorator, which lets us marks tests that we know should produce an exception:
In[45]:
import nose@nose.tools.raises(TypeError, ValueError)def test_raises_type_error(): raise TypeError("This test passes")
In[46]:
test_raises_type_error()
In[47]:
@nose.tools.raises(Exception)def test_that_fails_by_passing(): pass
In[48]:
test_that_fails_by_passing()
---------------------------------------------------------------------------AssertionError Traceback (most recent call last)Cell In [48], line 1----> 1 test_that_fails_by_passing()File /opt/hostedtoolcache/Python/3.8.14/x64/lib/python3.8/site-packages/nose/tools/nontrivial.py:67, in raises.<locals>.decorate.<locals>.newfunc(*arg, **kw) 65 else: 66 message = "%s() did not raise %s" % (name, valid)---> 67 raise AssertionError(message)AssertionError: test_that_fails_by_passing() did not raise Exception
We could reimplement this ourselves now too, using the context manager we wrote above:
In[49]:
def homemade_raises_decorator(exception): def wrap_function(func): # Closure over exception # Define a function which runs another function under our "raises" context: def _output(*args): # Closure over func and exception with reimplement_raises(exception): func(*args) # Return it return _output return wrap_function
In[50]:
@homemade_raises_decorator(TypeError)def test_raises_type_error(): raise TypeError("This test passes")
In[51]:
test_raises_type_error()
FAQs
Will UCL accept me with lower grades? ›
2.1.
All UCL programmes require GCSE or equivalent passes in English Language and Mathematics at grade 5 / C or higher.
Failure in more than 60 credits typically requires additional tuition to attain the academic standards required by UCL. If you fail in more than 60 credits you will therefore need to Repeat the modules with attendance and fees in the following academic session.
What if I fail in one module in UCL? ›A student who is required to Repeat must re-enrol on the failed modules in the following academic session.
How hard is it to get a UCL offer? ›Many of UCL's most competitive courses have a far lower acceptance rate, for example Fine Art (BFA), UCL's most competitive course, has an acceptance rate of around 4.7%. For extra support in applying for UCL's competitive courses, contact the Profs' admissions experts.
Can you appeal a UCL rejection? ›UCL decisions on applications are final, and there is no right of appeal against them. UCL will consider a complaint against any decision only if there is substantive evidence of an irregularity in the processing of the application in question.
What is a passing grade at UCL? ›| 1. | The Pass Mark at Levels 4, 5 and 6 (Undergraduate and Graduate Certificate/ Diploma level) must be 40.00% or Grade D. |
|---|---|
| 2. | The Pass Mark at Level 7 (Taught Postgraduate level) must be 50.00% or Grade C. |
Assignment briefs will include clear instructions about word counts, the inclusion of footnotes, diagrams, images, tables, figures and bibliographies etc. and you are expected to adhere to the requirements for each assessment. If you exceed these parameters you may receive a reduction in marks.
Are UCL exams open book? ›Open book assessments, such as Controlled Condition Exams or Take-Home Papers allow you to consult reference materials while you're taking the assessment. You will be completing this using your computer or tablet, most likely away from UCL – in your own space at home or your university accommodation.
Is it OK to fail a course at uni? ›No, it is not a sign to give up. It is a sign to party less and work more on academics. Many students fail a class somewhere along the way during university. It can actually be a helpful event in developing self-discipline.
What percentage of UCL students get a first? ›| Type of degree | UCL mark | Netherlands & Spain |
|---|---|---|
| 1st | 70-100% | 9 |
| 2:1 | 65-69% | 7 |
| 60-64% | ||
| 2:2 | 55-59% | 6 |
How many modules can you fail UK? ›
The general principle is that you are entitled to receive the teaching for a module once and that you can have two attempts at passing it.
How strict is UCL entry requirements? ›...
GCSE and equivalent qualifications
- 8 and 9 = A*
- 7 = A.
- 6 = B.
- 5 = C.
We aim to provide applicants with a decision within 10 weeks of their application being complete and will let you know either on your applicant portal or by email if we encounter any delays.
Can an average student get into UCL? ›Admission to UCL is competitive and meeting the minimum entry requirements does not guarantee admission. Please note that individual departments may seek grades at a higher level.
Do UCL give lower offers? ›Access UCL is our contextual offer scheme for applicants from groups that are underrepresented at UCL. Applicants that are eligible and successful in receiving an offer, will automatically receive an offer that is lower than the standard entry requirements for the programme.
What happens if no university accepts me? ›Universities don't have to give you a reason, but if the rejection has left you wondering you could contact the university to ask for feedback. Try not to take it too personally. Admissions tutors will be sifting through hundreds of applications and it might not always be obvious why it's not good news for you.
What if all universities reject me? ›If you've received multiple university rejections, don't worry. Anyone who has already applied to university through UCAS can use the free UCAS Extra service to continue applying for courses. You can even apply for a different subject to your original choice if you've had a change of heart.
Why do universities reject you? ›Your application does not meet the academic threshold
If it doesn't meet their requirement, they may think that you are not suited for the learning environment of the school. Apply to colleges where you have a good chance of getting in based on your grades and academic achievements.
C - this is a grade that rests right in the middle. C is anywhere between 70% and 79% D - this is still a passing grade, and it's between 59% and 69% F - this is a failing grade.
Is a score of 60% passing? ›In fact, a “D” is considered passing in both high school and college, as it's above 60%. While a passing grade may be as low as 60%, you will want to aim higher for many reasons. As a college student, you don't want to aim to barely pass a class.
Is 60 a pass in university? ›
...
Quebec.
| Letter | Percent |
|---|---|
| B | 75–79% |
| B− | 70–74% |
| C+ | 65–69% |
| C (passing grade) | 60–64% |
Students in Year 4, 5 and 6 of the MBBS Programme are eligible to receive an iPad to support teaching and learning whilst on Clinical Placements. Student can access a range of apps including: BMJ Best practice.
Does UCL have a high acceptance rate? ›The University College London has an acceptance rate is around 63%, making it moderately easier to get into. This means that out of 100 applicants, around 63 students seeking admission at the University College London, get accepted.
Does UCL look at as grades? ›Your GCSEs grades will be taken into consideration with the rest of your application, i.e. your predicted/achieved grades for your A levels (or equivalent qualifications), your personal statement, and your reference.
Are UCL lectures recorded? ›Lecturecast is UCL's lecture capture system. That goes beyond the simple recording and viewing of lectures, allowing staff to transform their content into an interactive experience and enabling students to engage further with pre-recorded lecture material.
Is open book exam hard? ›Online and open-book tests can be even worse. There are multiple reasons for this. Firstly, because the test questions tend to be more difficult, not only do you need to spend more time answering them, but any deficiencies in knowledge take disproportionately longer to fill.
Can you use your phone in an open book exam? ›Dictionaries should be free of all notes/annotations. Any dictionary found to have notes will be used as evidence of Academic Misconduct for which penalties apply. Please be advised that mobile phones or other electronic device detectors may be in use during examinations.
What happens if an international student fails a class? ›If you fail a required course, you will have to take it again. A grade of F-did not attend or F-did not take the final or W (withdrawal) indicate you dropped below full-time enrollment and puts you out of status.
Will I pass if I fail in one subject? ›If you fail one subject, you can make up for it in a compartment test, which is normally held within a few months. You will be declared failed that is if you fail two or more subjects, and you will have to repeat 12th grade. You have a chance of passing your academic if you pass the improvement test/exam.
How many times can you fail uni? ›Yes, it is possible to retake your final year at university but it's important to go in with the right mindset and you're prepared as you can generally only retake it once. While doing your retake all your grades will be capped at 40% and you would also have to pay a fee for each retake.
Is 75% a first at uni? ›
UK degree classifications are as follows: First-Class Honours (First or 1st) (70% and above) Upper Second-Class Honours (2:1, 2. i) (60-70%)
Is UCL a respected university? ›...
Times Higher Education World University Rankings 2018 - full top 200.
Third class
Commonly known as a “third”, this degree is the lowest level of honour's degree achievable. Very few graduates achieve a third-class honours.
you must repeat the modules that you failed. you may be excluded from further study and you may not continue.
Do I have to pass every module? ›Postgraduate students must normally pass all their modules to meet their award requirements. However, the relevant Examination Board may disregard failure in up to 30 credits where you have achieved a module mark of 40-49.9 in each of the failed modules and your overall average is 50.0 or greater.
What happens if an international student failed a class in UK? ›If your new course is shorter, the length of your visa could be reduced. If the course is longer you will need to apply to extend your visa before your current visa expires.. If you fail an exam: And you need repeat classes during the first semester you can remain in the UK and, if necessary, you can extend your visa.
Do universities accept retakes? ›Almost all universities, including Oxford and Cambridge, officially accept exam resits, meaning that there's no chance of being barred from applying. However, the university will be aware that you've achieved the grade by resitting, and so may affect your chances of receiving an offer.
What is UCL acceptance rate for international students? ›*Need assistance to study in UK? Y-Axis is here to assist you in all the ways. UCL has an acceptance rate of 48% and to get admission in it, students need to get a Minimum GPA of 3.6 out of 4, which is equivalent to 87% to 89%, and an IELTS score of at least 6.5.
What subjects is UCL best for? ›UCL currently has nine subject areas ranked in the global top 10, including Education (1), Architecture & the Built Environment (3), Archaeology (3), Anatomy & Physiology (6), Anthropology (4), Geography (9), Medicine (7), Pharmacy & Pharmacology (4), and Psychology (7).
Do universities accept lower grades than offer? ›Most universities that have course vacancies during Clearing will be prepared to accept you if your grades are below their entry requirements as long as you sound passionate and are right for the degree subject. They may also accept you based on the UCAS points you've accumulated rather than you final grades.
Can I apply to a uni if my predicted grades are lower? ›
Universities will consider slightly lower predicted grades for most degree programmes. This is because they know your predicted grades might not be an accurate reflection of your abilities and your final results could be higher. It's important not to be disheartened if your predicted grades are lower than you'd hoped.
Can you get into university with low marks? ›The short answer is yes. I've worked with a number of students who had bad grades in high school, but went on to do well in college. I've also worked with a few, however, who weren't quite ready for prime time. So while the answer is yes, a student with bad grades can still go to college.
Does everyone get an interview at UCL? ›Only a small number of UCL programmes ask applicants to complete an interview or test as part of their application. If we need you to do this, someone from the relevant academic department will contact you with further details. If you have any questions you should respond to the sender of the email.
Can I study in UK with low grades? ›If you wish to study in the UK but have a low exam score or missed out by one grade point you can apply for Clearing at UK universities. The UCAS Clearing process for September 2021 opened on 05 July 2021 and SI-UK India's dedicated Clearing team can help you apply.
Can I study abroad with low marks? ›Answer: Yes it is possible to study abroad even if you don't have so good grades. Language schools abroad accept students on any level, and there are also other schools like American Community Colleges that accept students with lower grades.
What is the lowest grade you can get in the UK? ›There are recent changes in the GCSE grading system in England, the 9-1 grading system was implemented. These subjects grades will be graded between 9-1, the highest grade is 9, while 1 is the lowest, not including a U (ungraded).
What is the lowest grade to pass a college? ›What is a passing grade? Many college grading systems consider a D, or 65 percent, to be the lowest passing grade. Note that different schools, programs, or classes may have different cutoff points for what they consider a passing grade.
How many students get their predicted grades? ›Predicted grades are infamously inaccurate. Only 16 per cent of students actually achieve them, according to research by University College London; in 2019, 84 per cent of predicted grades were wrong, with 79 per cent of them amounting to overestimates.
Can I get into uni with 120 UCAS points? ›According to reports from UCAS, most business-related courses require BBB at A-Level which brings a total of 120 UCAS points. On the other hand, law related courses will usually ask for ABB at A-Level which amounts to 128 UCAS points.
Is 49 a fail in university? ›If you score below 49% for an exam, you do not pass that module, so anything below a D symbol at university level is considered a fail. In the instance that you do not pass an exam but fall within the bracket of almost passing (usually obtaining 45%+), you will be eligible to write a supplementary exam.
Is 40 percent a pass at uni? ›
Marks and Grades
The pass mark is 40% and it is relatively unusual for students to regularly achieve marks of 70% or above (in fact, only 10% of students receive marks this high).