We have Generators & Threading in Python also. A person from Java/Scala background can consider Generator like Lazy evaluation done in these languages. But obviously all these features have to be used in particular scenario only & when required else you will be making a mess of your code.
One imaginary scenario I tried to code where we need Fibonacci numbers, & suppose this process takes time to generate each such number. Then I need to multiple each such fibonacci number with 2 which suppose again takes time.
If we first generate all the Fibonacci numbers then supply those numbers further to multiply with 2, then we have to wait for enough to see the first result even. It will also take the memory to hold all those numbers, but I am not considering that for now as we have enough memory nowadays. But it will take time to get the first result even.
While if I use Generator then I make generation of Fibonacci number on demand & I use Threads to multiply the fibonacci number with 2 concurrently. Though I could have used pool of threads here for better code.
It is not a quality code but it should be sufficient to get the idea, I am trying to convey here.
But one can see the impovement after usage of Generator & Threading here as shown below -
One imaginary scenario I tried to code where we need Fibonacci numbers, & suppose this process takes time to generate each such number. Then I need to multiple each such fibonacci number with 2 which suppose again takes time.
If we first generate all the Fibonacci numbers then supply those numbers further to multiply with 2, then we have to wait for enough to see the first result even. It will also take the memory to hold all those numbers, but I am not considering that for now as we have enough memory nowadays. But it will take time to get the first result even.
While if I use Generator then I make generation of Fibonacci number on demand & I use Threads to multiply the fibonacci number with 2 concurrently. Though I could have used pool of threads here for better code.
It is not a quality code but it should be sufficient to get the idea, I am trying to convey here.
But one can see the impovement after usage of Generator & Threading here as shown below -
from threading import Thread, Lock
import sys
import time
def seqHeavyMethod(n):
# Does some heavy calculation
# and time taking task
res = n*2
time.sleep(1)
print(n, res)
message = str(n)+':'+str(res)
fileName = 'output.txt'
fileWrite(fileName, message, None)
def heavyMethod(n, lock):
# Does some heavy calculation
# and time taking task
with lock:
time.sleep(1)
res = n * 2
print(n, res)
message = str(n)+':'+str(res)
fileName='output.txt'
Thread(target=fileWrite, args=(fileName, message, lock)).start()
def fileWrite(fileName, message, lock):
if lock is not None:
with lock:
with open(fileName, 'a') as file:
file.write(message+'\n')
else:
with open(fileName, 'a') as file:
file.write(message + '\n')
# Generates the Fibonacci numbers and store them in the list to return
def listCreate(n):
ls = []
a, b = 0, 1
time.sleep(2)
while a < n:
ls.append(a)
# Suppose it is also time taking step
time.sleep(2)
a, b = b, a + b
return ls
# Generates the fibonacci numbers for the given numbers
def customGenerator(n):
a, b = 0, 1
time.sleep(2)
yield a
while a < n:
# Suppose it is also time taking step
time.sleep(2)
a, b = b, a+b
yield a
val = -1
# A method is created to get the next value from the Generator
def fetcher(gen, n):
global val
try :
for _ in range(n):
val = next(gen)
# print(val)
except:
val = -5
def threadedCall(n):
start = time.time()
global val
last = -2
gen = customGenerator(n)
t = Thread(target=fetcher, args=(gen, n))
t.start()
processed = True
lock = Lock()
while True:
if val > -1 and val > last:
Thread(target=heavyMethod, args=(val, lock)).start()
last = val
if val == 1 and val == last and processed:
Thread(target=heavyMethod, args=(val, lock)).start()
processed = False
if val == -5:
break
totalTime = time.time() - start
size = sys.getsizeof(val)+sys.getsizeof(last)
print('Time taken via Threading : ', totalTime)
print('Space taken by threads approach : ', size)
message = 'Time taken via Threading approach : ' + str(totalTime)
fileName = 'output.txt'
fileWrite(fileName, message, None)
message = 'Space taken by Threading approach : ' + str(size)
fileWrite(fileName, message, None)
def seqApproach(n):
start = time.time()
ls = listCreate(n)
for item in ls:
seqHeavyMethod(item)
totalTime = time.time() - start
size = sys.getsizeof(ls)
print('Time taken via Sequential approach : ', totalTime)
print('Space taken by Sequential approach : ', size)
message = 'Time taken via Sequential approach : ' + str(totalTime)
fileName = 'output.txt'
fileWrite(fileName, message, None)
message = 'Space taken by Sequential approach : ' + str(size)
fileWrite(fileName, message, None)
def Main():
n = 30
threadedCall(n)
seqApproach(n)
if __name__=='__main__':
Main()
import sys
import time
def seqHeavyMethod(n):
# Does some heavy calculation
# and time taking task
res = n*2
time.sleep(1)
print(n, res)
message = str(n)+':'+str(res)
fileName = 'output.txt'
fileWrite(fileName, message, None)
def heavyMethod(n, lock):
# Does some heavy calculation
# and time taking task
with lock:
time.sleep(1)
res = n * 2
print(n, res)
message = str(n)+':'+str(res)
fileName='output.txt'
Thread(target=fileWrite, args=(fileName, message, lock)).start()
def fileWrite(fileName, message, lock):
if lock is not None:
with lock:
with open(fileName, 'a') as file:
file.write(message+'\n')
else:
with open(fileName, 'a') as file:
file.write(message + '\n')
# Generates the Fibonacci numbers and store them in the list to return
def listCreate(n):
ls = []
a, b = 0, 1
time.sleep(2)
while a < n:
ls.append(a)
# Suppose it is also time taking step
time.sleep(2)
a, b = b, a + b
return ls
# Generates the fibonacci numbers for the given numbers
def customGenerator(n):
a, b = 0, 1
time.sleep(2)
yield a
while a < n:
# Suppose it is also time taking step
time.sleep(2)
a, b = b, a+b
yield a
val = -1
# A method is created to get the next value from the Generator
def fetcher(gen, n):
global val
try :
for _ in range(n):
val = next(gen)
# print(val)
except:
val = -5
def threadedCall(n):
start = time.time()
global val
last = -2
gen = customGenerator(n)
t = Thread(target=fetcher, args=(gen, n))
t.start()
processed = True
lock = Lock()
while True:
if val > -1 and val > last:
Thread(target=heavyMethod, args=(val, lock)).start()
last = val
if val == 1 and val == last and processed:
Thread(target=heavyMethod, args=(val, lock)).start()
processed = False
if val == -5:
break
totalTime = time.time() - start
size = sys.getsizeof(val)+sys.getsizeof(last)
print('Time taken via Threading : ', totalTime)
print('Space taken by threads approach : ', size)
message = 'Time taken via Threading approach : ' + str(totalTime)
fileName = 'output.txt'
fileWrite(fileName, message, None)
message = 'Space taken by Threading approach : ' + str(size)
fileWrite(fileName, message, None)
def seqApproach(n):
start = time.time()
ls = listCreate(n)
for item in ls:
seqHeavyMethod(item)
totalTime = time.time() - start
size = sys.getsizeof(ls)
print('Time taken via Sequential approach : ', totalTime)
print('Space taken by Sequential approach : ', size)
message = 'Time taken via Sequential approach : ' + str(totalTime)
fileName = 'output.txt'
fileWrite(fileName, message, None)
message = 'Space taken by Sequential approach : ' + str(size)
fileWrite(fileName, message, None)
def Main():
n = 30
threadedCall(n)
seqApproach(n)
if __name__=='__main__':
Main()
Output you will see as same but the response & the space taken matter here.
Below is the output of above code & you will the 'output.txt' file created in the same directory -
Below is the output of above code & you will the 'output.txt' file created in the same directory -
