[Update 8/2018] I picked this back up after a break and am aiming for Python 3.8. See this project and PEP 554. <fingers_crossed/>
Contents:
Backstory
The Proposal
The Outcome
The Details
Backstory
For the longest time I've heard the argument that multi-processing is the Pythonic way to get multi-core. Threads are an anti-pattern, some say. They are a thing only because Microsoft made them a thing, some say. I've heard anecdotal evidence from people that actually doparallel (not just concurrent) computing (e.g. scientific, render farms) that they favor multi-processing approaches, especially because they typically go multi-host anyway.
You could also argue that in practice nearly all concurrent programming is IO-bound and that asyncio solves that for us. Recent-ish discussions (e.g. PEP 492) lead me to believe that it's not nearly that simple.
Ultimately the merits of multi-processing and asyncio over multi-core threading, regardless of whether or not a valid or sufficient argument, do not mitigate the popular and pervading *sentiment* that Python is weak when it comes to leveraging multi-core and handling concurrency (or more accurately computational parallelism). And perception ""is 9/10th of the law"" (or arguably higher).
Folks looking for a solution are going to search for one that matches the model in their brain. Much like with organic molecules, if the conceptual bind points don't line up they aren't going to connect with what Python is offering. The power of Python is that it maps well onto our brains. Though concurrency/parallelism isn't very suited to our brains, it is one key place where Python doesn't do a good job of matching conceptual expectations at large.
The Proposal
In short, Python's multi-core story is murky at best. Not only can we be more clear on the matter, we can improve Python's support. The result of any effort must make multi-core (and concurrency/parallelism) support in Python obvious, unmistakable, and undeniable (and keep it Pythonic).Early in 2015 I'd reached my limits with all the criticisms and misunderstandings. So in the spirit of open source I resolved to do something about it. Since this was not an area of expertise for me I did a lot of reading and reached out to experts I know (thanks to Guido, Nick, Sarah, Graham, and others). In a few months I felt like I had a good enough understanding and a good solution.
In June of 2015 I introduced my solution on the python-ideas mailing list. The gist is to use CPython's existing subinterpreters to isolate GIL-free execution threads with a CSP front end. My hope was to finish the first stage of work in time for Python 3.6 (i.e. right about now). The reception was generally positive. There was even discussion on reddit. I was encouraged.
The Outcome
Going in I knew it would be a challenging project. However, the solution I proposed was tractable in the desired time frame, building on a lot of existing parts and decomposing into manageable stages. The blockers were well understood, mainly involving subinterpreter bugs and PEP 432. I also received several solid offers for help. In October I even went to PyCon UK to coordinate some of the efforts.At the same time it became clear that my life was getting too busy to make much progress. In early 2016 I decided to table the project. I wasn't giving up yet and hoped to get back to it. Furthermore, there were also many parts of the project that stand on there own as useful features. That's about where things are at right now.
The Details
This is where I try to summarize my proposal and relevant information.Summary
- expose subinterpreters in Python (a low-level stdlib module)
- support passing objects between subinterpreters
- add subinterpreter serial-execution mode
- add a high-level CSP module to the stdlib
Phases
- resolve blockers
- add "interpreters" module
- minimal multi-core solution
- subinterpreter "serial execution mode" (no GIL )
- channels supporting immutable objects
- no extension modules
- csp module
- expanded support
- support more types in channels
- performance optimization
- extension module support (PEP 489 compliant only)
Requirements
- "make multi-core support in Python obvious, unmistakable, and undeniable (and keep it Pythonic)"
- no significant impact on single-threaded performance
- maintain backward compatibility (C-API, etc.)
- (pseudo-)compatibility with multiprocessing/threading/concurrent.futures APIs
- a multi-core concurrency model/approach that fits our brains
- Python APIs
- supportable on other Python implementations
Blockers
- PEP 432 ("Simplifying the CPython startup sequence")
- improved interpreter isolation
- (mostly) eliminate process globals
- handling of process state (env vars, etc.)
- subinterpreter bugs:
- http://bugs.python.org/issue4202
- http://bugs.python.org/issue6531
- http://bugs.python.org/issue10915
- http://bugs.python.org/issue15751
- http://bugs.python.org/issue21387
- http://bugs.python.org/issue24553
- http://bugs.python.org/issue24554
- http://bugs.python.org/issue24932
- https://github.com/pyca/cryptography/issues/2299
- https://github.com/GrahamDumpleton/mod_wsgi/issues/43
Standalone Improvements
- faster/cleaner interpreter startup?
- better multiple-interpreters-per-process support
- named subinterpreters
- interpreters module (a la threading)
- _interpreters module (a la _threading)
- more efficient sharing between interpreters (e.g. builtins)
- faster/cleaner subinterpreter startup
- share some modules
- leverage object sharing
- refactor C-API to take interpreter arg
- leaner subinterpreters?
- refcounts in own memory page
- factor out pickle-independent parts of multiprocessing
- better object immutability
- truer immutability?
- immutable mode?
- frozen objects
- issue #24991: Define instance mutability explicitly on type objects
- several PEPs
- isolated object graphs
- "isolated" object
- memory model (all in same page)
- related to RDM project
Specific Additions
- channels (a la queue)
- object sharing between interpreters
- "immutable objects: int, float, str, tuple, bool, frozenset, complex, bytes, None
- - containers (tuple, frozenset) must hold only immutable objects"
- types that implement __shared__
- "frozen" objects
- read-only views
- "owned" objects (transfer ownership)
- C channels
- in own module?
- PEP-489 slot for subinterpreter support?
- subinterpreter "serial execution mode"
- add mode management
- start each in own thread
- disallow threading
- disallow forking
- eliminate GIL within each subinterpreter
- csp module
- inspired by python-csp
- shared-nothing "thread" concurrency model
- uses subinterpreters in serial execution mode by default
- object ownership
- "Local Interpreter Lock"
Specific Changes
- drop GIL between interpreters?
Python Alternatives
- threading
- multiprocessing
- asyncio
- STM (Armin Rigo, PyPy)
- pyparallel (Trent Nelson)
- dask
- gilectomy (Larry Hastings, CPython)
- otherwise remove the GIL
- better concurrency primitives for threaded programming
- add multi-core support to the asyncio event loop
- better documentation
- Jython
- IronPython
- other Python implementations
- fibers
- do nothing
This is great, will it be possible to have subinterpreters with different python binaries (e.g. [cpython <-> [jython] ] ) ?
ReplyDeleteI am glad that I saw this post. It is informative blog for us and we need this type of blog thanks for share this blog, Keep posting such instructional blogs and I am looking forward for your future posts. Python Projects for Students Data analytics is the study of dissecting crude data so as to make decisions about that data. Data analytics advances and procedures are generally utilized in business ventures to empower associations to settle on progressively Python Training in Chennai educated business choices. In the present worldwide commercial center, it isn't sufficient to assemble data and do the math; you should realize how to apply that data to genuine situations such that will affect conduct. In the program you will initially gain proficiency with the specialized skills, including R and Python dialects most usually utilized in data analytics programming and usage; Python Training in Chennai at that point center around the commonsense application, in view of genuine business issues in a scope of industry segments, for example, wellbeing, promoting and account. Project Center in Chennai
DeleteI like your blog, I read this blog please update more content on python, further check it once at python online training
ReplyDeleteVery informative Blog!! Thanks for sharing...
ReplyDeleteDjango Online Courses
Django Training in Hyderabad
Python Django Training in Hyderabad
Python Django Online Training
Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
ReplyDeletetop servicenow online training
Poker online situs terbaik yang kini dapat dimainkan seperti Bandar Poker yang menyediakan beberapa situs lainnya seperti http://62.171.128.49/hondaqq/ , kemudian http://62.171.128.49/gesitqq/, http://62.171.128.49/gelangqq/, dan http://62.171.128.49/seniqq. yang paling akhir yaitu http://62.171.128.49/pokerwalet/. Jangan lupa mendaftar di panenqq silakan dicoba ya boss
ReplyDelete
ReplyDeleteHello, I have gone through your post Its really awesome.Thats a great article. I am also want to share about python online course and advanced python training. thank you
Thank you for sharing your thoughts and knowledge on this topic.
ReplyDeletePython Online Training in Hyderabad
Python Institute in Hyderabad
Python Course in Hyderabad
your post Its really fantastic.
ReplyDeletePHP Training in Chennai | Certification | Online Training Course | Machine Learning Training in Chennai | Certification | Online Training Course | iOT Training in Chennai | Certification | Online Training Course | Blockchain Training in Chennai | Certification | Online Training Course | Open Stack Training in Chennai |
Certification | Online Training Course
Very interesting post, I get great knowledge from this post.
ReplyDeletethanks for shearing with us.
GMAT online Training
GMAT coaching in hyderabad
Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
ReplyDeleteakilmanati
I feel really happy to have seen your webpage.I am feeling grateful to read this.you gave a nice information for us.please updating more stuff content...keep up!!
ReplyDeleteData Science Training In Chennai
Data Science Online Training In Chennai
Data Science Training In Bangalore
Data Science Training In Hyderabad
Data Science Training In Coimbatore
Data Science Training
Data Science Online Training
Full Stack Course Chennai
ReplyDeleteFull Stack Training in Bangalore
Full Stack Course in Bangalore
Full Stack Training in Hyderabad
Full Stack Course in Hyderabad
Full Stack Training
Full Stack Course
Full Stack Online Training
Full Stack Online Course
Nice post. I was checking constantly this blog and I’m impressed! Extremely useful information specially the last part 🙂 I care for such info much. I was looking for this particular information for a long time. Thank you and best of luck.
ReplyDeleteJava Training in Chennai
Java Training in Bangalore
Java Training in Hyderabad
Java Training
Java Training in Coimbatore
thank you for the information
ReplyDeleteangular js course in chennai
angular course in chennai
angular js online course in chennai
angular js course in bangalore
angular js course in hyderabad
angular js course in coimbatore
angular js course
angular js online course
Very useful information, the post shared was very nice.
ReplyDeletepython Online Training
Thank you for sharing.
ReplyDeleteData Science Online Training
Salesforce Online Training
Mua vé tại Aivivu, tham khảo
ReplyDeletevé máy bay đi Mỹ giá rẻ 2021
vé máy bay giá rẻ vinh sài gòn
vé máy bay nha trang - hà nội
vé máy bay đi đà lạt khứ hồi
vé máy bay từ los angeles về việt nam
dịch vụ taxi sân bay nội bài
combo quy nhơn phú yên
rastgele görüntülü konuşma - kredi hesaplama - instagram video indir - instagram takipçi satın al - instagram takipçi satın al - tiktok takipçi satın al - instagram takipçi satın al - instagram beğeni satın al - instagram takipçi satın al - instagram takipçi satın al - instagram takipçi satın al - instagram takipçi satın al - binance güvenilir mi - binance güvenilir mi - binance güvenilir mi - binance güvenilir mi - instagram beğeni satın al - instagram beğeni satın al - polen filtresi - google haritalara yer ekleme - btcturk güvenilir mi - binance hesap açma - kuşadası kiralık villa - tiktok izlenme satın al - instagram takipçi satın al - sms onay - paribu sahibi - binance sahibi - btcturk sahibi - paribu ne zaman kuruldu - binance ne zaman kuruldu - btcturk ne zaman kuruldu - youtube izlenme satın al - torrent oyun - google haritalara yer ekleme - altyapısız internet - bedava internet - no deposit bonus forex - erkek spor ayakkabı - webturkey.net - karfiltre.com - tiktok jeton hilesi - tiktok beğeni satın al - microsoft word indir - misli indir
ReplyDeleteSitus Nonton movie, film dan tv series terbaru dengan subtitle indonesia diupdate setiap hari, dari situs terpopuler nonton disini link
ReplyDeletedi bawah ini
layarkaca21
bioskopkeren
daymovie
terbit21
boomthis
Situs judi online terpercaya
http://199.188.201.133
http://162.213.251.13
bonekaqq
indoqq99
sahabatqq
Tamamen Otomatik Sistem ile Siparişleriniz 7 Gün 24 Saat Hızlı ve Sorunsuz Bir Şekilde Tamamlanmaktadır. instagram takipçi satın al ve daha fazlası.
ReplyDeleteinstagram takipçi satın al
instagram beğeni satın al
instagram takipçi satın al
instagram takipçi satın al
instagram takipçi satın al
instagram takipçi satın al
instagram takipçi satın al
takipçi satın al
ucuz takipçi satın al
tiktok takipçi satın al
takipçi satın al
ReplyDeleteinstagram takipçi satın al
https://www.takipcikenti.com
marsbahis
ReplyDeletebetboo
sultanbet
marsbahis
betboo
sultanbet
ras handwritten notes in hindi
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteNice article with valuable information. Thanks for sharing.
ReplyDeletePython Online Training
Artificial Intelligence Online Training
Data Science Online Training
Machine Learning Online Training
AWS Online Training
UiPath Online Training
This comment has been removed by the author.
ReplyDeleteAivivu, đặt vé máy bay tham khảo
ReplyDeletemua ve may bay di my
khi nào có chuyến bay từ mỹ về việt nam
chuyến bay thương mại từ canada về việt nam
tình hình vé máy bay từ nhật về việt nam
gia ve may bay vietjet tu han quoc ve viet nam
Vé máy bay từ Đài Loan về VN
ve may bay chuyen gia nuoc ngoai sang Viet Nam
Webimiz ile yüksek ödeme oranları casinolar oynabilirsiniz. Venüsbet Venüsbet Grandbetting Grandbetting Hiltonbet Betnano Marsbahis Meritking Asyabahis
ReplyDeleteeskişehir
ReplyDeleteizmir
muğla
yalova
çanakkale
düzce
antalya
aydın
mate
tiktok jeton hilesi
ReplyDeletetiktok jeton hilesi
binance referans kimliği
gate güvenilir mi
tiktok jeton hilesi
paribu
btcturk
bitcoin nasıl alınır
This blog covers some most asked topics in the Azure Data Engineer Interview Questions with fully explained answers and details dp203
ReplyDeleteMicrosoft recently updated a certification named Azure Data Engineer Associate. To get this tag you need to clear one examination named: Exam DP-203: Data Engineering on Microsoft Azure.
ReplyDeleteAimore Technologies is the best Selenium training institute in Chennai with 6+ years of experience. We are offering online and classroom training.
ReplyDeleteVisit Us: Selenium Training in Chennai
İnstagram takipçi satın al! İnstagram takipçi sitesi ile takipçi satın al sende sosyal medyada fenomen olmaya bir adım at. Sende hemen instagram takipçi satın almak istiyorsan tıkla:
ReplyDelete1- takipçi satın al
2- takipçi satın al
3- takipçi satın al
This article is very appealing to thinking people like me.
ReplyDeleteIt’s not only thought-provoking, it draws you in from the beginning.
This is well-written content. The views here are also appealing to me.
Thank you. dg grand.만남사이트
Ahaa, its nice dialogue about this piece of writing at this place at this webpage,
ReplyDeleteI have read all that, so at this time me also commenting here.
havanese dogs for sale
havanese puppies for sale
teacup havanese puppy
chocolate havanese puppy for sale
teacup pomeranian for sale
doodle puppies
aussiedoodle puppies for sale
bernedoodle puppies for sale
goldendoodle puppies for sale
yorkie puppies for sale
https://www.fluffyhavanese.com/
Thanks for sharing your thoughts. I truly appreciate your efforts and I will be waiting for your further
ReplyDeletewrite ups thanks once again
pomeranian puppies for sales
pomeranian for sale
teacup pomeranian for sale near me
pomeranians for sale near me
teacup pomeranian for sale
pomeranian puppies for sales
pomeranian for sale
teacup pomeranian for sale
havanese puppies for sale near me
https://www.pomeranianpuppiesforsales.com/
I used to be suggested this website by means of my cousin. I’m now not sure whether
ReplyDeletethis submit is written by him as nobody else know such detailed approximately my problem.
ragdoll kitten for sale
ragdoll kittens for sale
mini goldendoodle for sale
mini bernedoodle puppies for sale
mini aussiedoodles for sale
havanese puppy for sale
teacup havanese puppies for sale
havanese puppies for sale near me
havanese puppies for sale under $1,000
havanese puppy for sale
https://thegorgeousragdolls.com/
At this time I am going away to do my breakfast, afterward having my breakfast coming over again to read further news
ReplyDeleteragdoll cat for sale
ragdoll cats for sale
aussiedoodle for sale
bernedoodles for sale
havanese puppies for sale under $1,000
teacup pomeranians for sale near me
teacup pomeranian puppies for sale $500
pomeranian puppies for sale near me
pomeranian for sale near me
https://thegorgeousdoodles.com/
I take pleasure in, result in I discovered exactly what I used to be taking a look
ReplyDeletefor. You’ve ended my 4 day lengthy hunt! God Bless you man.
ragdoll kittens for sale near me
ragdoll kitten near me
goldendoodle for sale
teacup havanese puppies for sale
pomeranian teacup for sale
havanese puppies for sale near me
havanese puppies for sale near me
teacup havanese puppy
chocolate havanese puppy for sale
Great Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us and I never get bored while reading your article because, they are becomes a more and more interesting from the starting lines until the end. best micronutrients for plants
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteThank you for sharing such a nice and interesting blog and really very helpful article. If you want to become Oracle Fusion SCM Certified Consultant, attend this Best Oracle SCM Training Course offered by the Unogeeks
ReplyDeleteinstagram takipçi satın al
ReplyDeleteشركة المثالية للتنظيف بالخبر
ReplyDeleteشركة المثالية للتنظيف بالقطيف
https://haileypets.com/teacup-shih-tzu-for-sale/
ReplyDeletehttps://haileypets.com/schnauser-puppies-for-sale/
https://furkittens.com/cat-persian-for-sale/