Wow, this is cool. Kenneth has started to identify and call out other libraries that are "barriers to entry" in Python. If this is a call to arms to scale the Requests experience, it could grow into something really powerful - a movement to systematically reinvent and rejuvenate the Python standard library.
Are you sure these are all the same? I can't check from here (on my mobile, poor reception) but the URLs look significantly different. This one is a slide deck, one of the ones you quote is a video.
Slightly off topic: in an effort to become a better programmer, and to improve the Python ecosystem, I've tried to write a "for Humans" style logging library. Would anyone mind giving me feedback?
- Why is your example logger named d? It might seem nitpicky but it's hard
to read an example with a meaningless single character variable.
- "d.close() # stop logging" - what is this? What does it mean to "stop
logging" and why do I want to?
- "Add a coroutine" - I would wager the lay Python developer doesn't even
know what this word means.
- lggr.Lggr() - Why not just name the class Logger?
- The default format variables are inconsistent about when words are
separated with an underscore.
- I can't make sense of the example logging calls. In one you pass a
dictionary as the second argument, in another you pass three strings as
separate arguments. Why would anyone pass a message like this instead of
just using standard string formatting? Especially when there's other
legit arguments like extra. It's not even really clear why you'd want to
pass something in extra instead of in the message.
I fully support your goal, but at a glance this seems like a confusing alternative
to logging.
EDIT: hot dang it's hard to format a bulleted list on HN
> Why is your example logger named d? It might seem nitpicky but
> it's hard to read an example with a meaningless single character variable.
No particular reason -- you're right, I will change this now.
> "d.close() # stop logging" - what is this? What does it mean to "stop logging"
> and why do I want to?
This is a way to "close" the logger and clean up all of its associated coroutines. An example of when you might want to is upon catching a fatal error to your program, you might want to log the error and then safely clean up all open files or network sockets to which the logger is writing.
The method name is clearly confusing -- what do you think of using "shutdown" instead?
> "Add a coroutine" - I would wager the lay Python developer doesn't
> even know what this word means.
Maybe not, but they should! The readme now includes a link to dabeaz's coroutines page, and I'll add a quick overview in a couple of minutes.
> lggr.Lggr() - Why not just name the class Logger?
For consistency's sake. Maybe if I had to start again I would call the project `logger`, but I decided to be "hip" and use a vowel-less name instead :)
> The default format variables are inconsistent about when words are separated
> with an underscore.
The initial idea was to mimic the variable names from the default logging module (http://docs.python.org/2/library/logging.html#logrecord-attr...). You're right that it is confusing though! I think I will rename everything to be lower case, one word, instead of the default module's mix of camelCase and underscore_separated names. Thoughts?
> I can't make sense of the example logging calls. In one you pass a
> dictionary as the second argument, in another you pass three strings
> as separate arguments. Why would anyone pass a message like this
> instead of just using standard string formatting? Especially when
> there's other legit arguments like extra. It's not even really clear
> why you'd want to pass something in extra instead of in the message.
I should definitely clarify what formats are allowed and aren't. The log message format is using standard string formatting -- string.format(), to be exact. The 'extra' argument is a result of trying to imitate the default library (see http://docs.python.org/2/library/logging.html#logging.Logger... for a description of the 'extra' kwarg), and can be useful when you'd like to pass information to every single log message.
I am fairly sure that most of the streams and sockets would get closed when the interpreter exited regardless.
Also, in accordance to the principle of "explicit is better than implicit," it's generally better to avoid messing with global state (such as atexit) behind the scenes, and to provide a way to shut down the logs manually.
your library has no tests! it's a good effort but I honestly wouldn't use it yet.
a few nit picks: i can't imagine a scenario where i'd want my logger to close stdout (or any other file descriptor) for me.
there is a lot of missing error handling, which is really important for something critical like a logger. what happens when disk space runs out? no timeouts on network operations?
also your SMTP and Gmail loggers don't form valid MIME messages (I can't log non-ascii?). you also seem to just swallow exceptions which is totally not what I would want or expect from a logging library.
keep at it though, the logging module's API (inspired by log4j) is fairly painful.
> there is a lot of missing error handling, which is really
> important for something critical like a logger. what
> happens when disk space runs out? no timeouts on
> network operations?
> [...]
> you also seem to just swallow exceptions which is totally
> not what I would want or expect from a logging library.
Lggr will fail silently by default, because your logging library shouldn't cause your
code to crash. If you'd like, it's quite easy to stop it from suppressing errors. For
more complex error handling, I think users should write their own coroutines for
handling log messages -- the default file printing and network sending coroutines
are merely the most basic case.
> also your SMTP and Gmail loggers don't form valid MIME messages
> (I can't log non-ascii?).
Please help me and add loggers that do format valid MIME message! The included loggers/coroutines
are just a few examples I came up with, but I'd love if other people were to contribute more :)
> keep at it though, the logging module's API (inspired by log4j)
> is fairly painful.
I like the idea of delegating logging to its own process in theory (in practice, it would depend on your configuration, I guess, it may mean a lot of context switching), but your built-in coroutines seem to rely on yield instead. Have you experimented with all three styles?
A big feature missing, IMHO, is file-based configuration. You definitely want this for larger projects.
Reminds me of a chat session I had with a dev at another office a few days ago:
him: "have you ever used urllib2"
me: "not if I can avoid it"
him: "I've just spent 3 days trying to do X"
me: "have you heard of Requests"
him: "nope"
...send URL...
him: "god damn it! Why didn't I talk to you three days ago"
20 minutes later he's replaced three days work with a dozen lines of easy to read Requests code.
Was there today at PyCon Canada and not only was the presentation Kenneth gave very good, but he was very awesome to talk to afterwards. I randomly came across his 500px account, so we sent him a shirt, and he wore it on Friday just to say thanks :3 (I actually didn't get to see him on the Friday night party, probably because I know him best by his pixelated twitter pic, but it is super cool that he take the time to think of stuff like that).
A bit off-topic, but I couldn't find this and was very curious: What font are the slides using in the code snippets? It looks very clean--I didn't realize it was monospace at first.
I was at his talk this morning at pycon canada. It was a very good talk, the highlight of the conference. I like his approach, although I think it'll be a challenge to get there since not everyone has his "visionary" and api building skills, but someone has to start somewhere. Well done.
Also there- loved it. I think the key is what he said about writing the readme first. Write the docs on how you'd want to use the software, then hold yourself to that API when you actually write it.
While I appreciate the effort to make some more simplistic APIs available alongside their more complex counterparts - libraries like urllib2 are not expendable in the process.
In the act of making an API simpler and abstracting the details - you loose a lot of the detailed control provided by these lower level libraries. Perhaps you won't miss the 10% as outlined in the slides, but someone out there will.
Do we really need both? I think no, but then again for most of what I do, urllib2 just works; I'm not really the target for requests. Just don't expect that requests will ever replace urllib2.
If requests hits the 90% and urllib2 the 10%, then requests should be the standard library and urllib2 a module. I think his point is more to point out good modules that most people should use, and should become common knowledge of what people should turn to when trying to do something. Java, for example, has several date and time options but Jodatime actually does what most people want to do with dates and times.
As much as I know everyone dislikes PHP, there's lots of people still working with it. I love Requests in Python, so I built Requests for PHP: http://requests.ryanmccue.info/
I have to agree with Kenneth's points, in that pragmatism should outweigh the theoretical points. For example, in PHP, using a class as a grouping for static methods is a bad idea. I agree to a point, but there's something to be said for the ability to do `Requests::get('http://google.com/)` versus all the crap you have to do with cURL. (Plus, it's always >90% test coverage, which I stick to religiously.)
urllib2 is so bad (and I know how to use it) that I have at times used subprocess + curl, which says a lot, because subprocess ain't exactly pretty either. But the thing is, urllib2 is included and is the devil I know.
I'm not sure why you feel this niche requirement requires three question marks. The point of modules like this is to make it easier to do the tasks that 95% of developers are likely to need to do regularly, at the cost of not covering all rare use cases.
It's a reasonable tradeoff for a specific library. But if he's trying to claim that the entire eco-system should put the API above all, then it's worth asking how the 5% use case integrates with this worldview. If I suddenly find I need multiple IPs, do I now need to switch my http client library?
No, you need to do what anyone who has fringe requirements has to do: patch the library yourself. Modify urllib3's (requests uses this as a go between for httplib/http.client) to accept/pass on the source_address that the HTTPConnection from httplib allows and then modify requests to accept/pass this on. This will take, at first glance, maybe half a day with some tests (likely much faster, but I'll allow that there might be something more involved than modifying urllib3's HttpConnectionPool and the accompanying changes for requests's Request model.
The slides don't mention this, but he did touch on having layered designs so the 5% remains possible. However, the primarily exposed api shouldn't get sacrificed for said features. By conceding on a feature that makes the API less usable, you sacrifice more than not having it in the first place (confusion, complexity). Even the GH Issue linked shows a possible solution to the issue using urllib2, which is just a layer down.
I have yet to ever have to use multiple outbound IPs from the same machine for an HTTP request from Python. It might be "fairly common" for whatever you do, but in terms of the general population, it's not.
Still, it's a fair point that it at least should be possible. It's fine by me if the uncommon 5% tasks are just as difficult to do in requests or even a little more so, so long as the most common 95% of tasks are a breeze.
So true... I often need to write something that will run on a Windows machine and I just end up having to use the old version of Beautiful Soup (before it became dependent on lxml). Pretty frustrating.
Thanks so much! Just tried it and the setup binaries worked like a charm! 20 seconds for the whole experience including downloading and testing. Very different from how I remember my experience with lxml on Windows when I tried it before! They should definitely have a link to this website from http://lxml.de/installation.html#ms-windows
requests & envoy have saved me a huge amount of hours. I wish I met Kenneth when he was in Toronto this weekend, I would have bought him anything he wanted.
