Tools for Pentesters. Compilation. Toxy. HTTP proxy. failure scenarios. ![]() The following distributions match your criteria (sorted by popularity): 1. Manjaro Linux (3) Manjaro Linux is a fast, user-friendly, desktop-oriented operating system. Download the latest Virtualbox for Mac OS X from https://www.virtualbox.org/wiki/Downloads and install it. For more details on VirtualBox installation, please visit. The Linux Data Science Virtual Machine is a CentOS-based Azure virtual machine that comes with a collection of pre-installed tools. These tools are commonly used for.It was mainly designed for fuzzing/evil testing purposes, when toxy becomes particularly useful to cover fault tolerance and resiliency capabilities of a system, especially in. Mit. M proxy among services. HTTP flow as you need, performing multiple evil actions in the middle of that process, such as limiting the bandwidth, delaying TCP packets, injecting network jitter latency or replying with a custom error or status code. It operates only at L7 (application level). It was built on top of. HTTP proxy, and it's also. Requires node. js +0. Full- featured HTTP/S proxy (backed by. Hackable and elegant programmatic API (inspired on connect/express). Admin HTTP API for external management and dynamic configuration. Featured built- in router with nested configuration. Hierarchical and composable poisoning with rule based filtering. Hierarchical middleware layer (both global and route scopes). Easily augmentable via middleware (based on connect/express middleware). Supports both incoming and outgoing traffic poisoning. Built- in poisons (bandwidth, error, abort, latency, slow read..). Rule- based poisoning (probabilistic, HTTP method, headers, body..). Supports third- party poisons and rules. Built- in balancer and traffic interceptor via middleware. Inherits API and features from. Compatible with connect/express (and most of their middleware). Able to run as standalone HTTP proxy. There're some other similar solutions like. Furthermore, the majority of the those solutions only operates at TCP L3 level stack instead of providing high- level abstractions to cover common requirements in the specific domain and nature of the HTTP L7 protocol, like toxy tries to provide. HTTP protocol primitives easily. Via its built- in hierarchical domain specific middleware layer you can easily augment toxy features to your own needs. HTTP transaction (e. One HTTP transaction can be poisoned by one or multiple poisons, and those poisons can be also configured to infect both global or route level traffic. HTTP request/response in order to determine, given a certain rules, if the HTTP transaction should be poisioned or not (e. Rules can be reused and applied to both incoming and outgoing traffic flows, including different scopes: global, route or poison level. Incoming request ) ↓. Toxy Router | ↓ - > Match the incoming request. Incoming phase | ↓ - > The proxy receives the request from the client. Exec Rules | | ↓ - > Apply configured rules for the incoming request. Exec Poisons | | ↓ - > If all rules passed, then poison the HTTP flow. HTTP dispatcher | ↓ - > Forward the HTTP traffic to the target server, either poisoned or not. Outgoing phase | ↓ - > Receives response from target server. Exec Rules | | ↓ - > Apply configured rules for the outgoing request. Exec Poisons | | ↓ - > If all rules passed, then poison the HTTP flow before send it to the client. Send to the client ) ↓ - > Finally, send the request to the client, either poisoned or not. Create a new toxy proxy. Default server to forward incoming traffic. Register global poisons and rules. Register multiple routes. Rule(rules. headers({'Authorization': /^Bearer (.*)$/i })). Infect outgoing traffic only (after the server replied properly). Poison(poisons. bandwidth({ bps: 5. Rule(rules. method('GET')). Rule(rules. time. Threshold({ duration: 1. Rule(rules. response. Status({ range: [ 2. Limit({ limit: 1. Rule(rules. method(['POST', 'PUT', 'DELETE'])). And use a different more permissive poison for GET requests. Limit({ limit: 5. Rule(rules. method('GET')). Handle the rest of the traffic. Close({ delay: 1. Read({ bps: 1. 28 })). Rule(rules. probability(5. Server listening on port: ', 3. Test it: ', 'http: //localhost: 3. Poisons host specific logic which intercepts and mutates, wraps, modify and/or cancel an HTTP transaction in the proxy server. Poisons can be applied to incoming or outgoing, or even both traffic flows. Poisons can be composed and reused for different HTTP scenarios. They are executed in FIFO order and asynchronously. Poisoning scopes. HTTP traffic received by the proxy server, regardless of the HTTP method or path. HTTP verb and URI path. Poisons can be plugged to both scopes, meaning you can operate with better accuracy and restrict the scope of the poisoning. Poisoning phases. Poisons can be plugged to incoming or outgoing traffic flows, or even both. This means, essentially, that you can plug in your poisons to infect the HTTP traffic. HTTP server or sent to the client. This allows you apply a better and more accurated poisoning based on the request or server response. For instance, given the nature of some poisons, like. Built- in poisons. Poisoning Phase. incoming / outgoing. Reaches the server. Infects the HTTP flow injecting a latency jitter in the response. Jitter value in miliseconds. Random jitter maximum value. Random jitter minimum value. Or alternatively using a random value. Inject response. Poisoning Phase. Reaches the server. Injects a custom response, intercepting the request before sending it to the target server. Useful to inject errors originated in the server. Response HTTP status code. Default. - Optional headers to send. Optional body data to send. It can be a. - Body encoding. Default to. toxy. Content- Type': 'application/json'}. Poisoning Phase. incoming / outgoing. Reaches the server. Limits the amount of bytes sent over the network in outgoing HTTP traffic for a specific time frame. This poison is basically an alias to. Amount of chunk of bytes to send. Default. - Packets time frame in miliseconds. Default. toxy. poison(toxy. Poisoning Phase. incoming / outgoing. Reaches the server. Limits the amount of requests received by the proxy in a specific threshold time frame. Designed to test API limits. Exposes typical. X- Rate. Limit- *. Note that this is very simple rate limit implementation, indeed limits are stored in- memory, therefore are completely volalite. There're a bunch of featured and consistent rate limiter implementations in. You might be also interested in. Total amount of requests. Default to. - Limit time frame in miliseconds. Default to. - Optional error message when limit is reached. HTTP status code when limit is reached. Default to. toxy. Limit({ limit: 5, threshold: 1. Poisoning Phase. Reaches the server. Reads incoming payload data packets slowly. Only valid for non- GET request. Packet chunk size in bytes. Default to. - Limit threshold time frame in miliseconds. Default to. toxy. Read({ chunk: 2. 04. Poisoning Phase. Reaches the server. Delays the HTTP connection ready state. Delay connection in miliseconds. Default to. toxy. Open({ delay: 2. 00. Poisoning Phase. incoming / outgoing. Reaches the server. Delays the HTTP connection close signal (EOF). Delay time in miliseconds. Default to. toxy. Close({ delay: 2. Poisoning Phase. incoming / outgoing. Reaches the server. Restricts the amount of packets sent over the network in a specific threshold time frame. Packet chunk size in bytes. Default to. - Data chunk delay time frame in miliseconds. Default to. toxy. Abort connection. Poisoning Phase. incoming / outgoing. Reaches the server. Aborts the TCP connection. From the low- level perspective, this will destroy the socket on the server, operating only at TCP level without sending any specific HTTP application level data. Aborts TCP connection after waiting the given miliseconds. Default to. , the connection will be aborted if the target server takes more than the. Default to. - Custom internal node. Default to. // Basic connection abort. Abort after a delay. In this case, the socket will be closed if. Poisoning Phase. incoming / outgoing. Reaches the server. Defines a response timeout. Useful when forward to potentially slow servers. Timeout limit in miliseconds. How to write poisons. Poisons are implemented as standalone middleware (like in connect/express). Here's a simple example of a server latency poison. Latency(delay) {. We name the function since toxy uses it as identifier to get/disable/remove it in the future. Latency(req, res, next) {. Timeout(clean, delay). Close). function on. Close() {. clear. Timeout(timeout). Listener('close', on. Close). var proxy = toxy(). Register and enable the poison. Latency(2. 00. 0)). You can optionally extend the build- in poisons with your own poisons. Poison(custom. Latency). Then you can use it as a built- in poison. Latency). For featured real example, take a look to the. Best Practices for Amazon Redshift Spectrum. Amazon Redshift Spectrum enables you to run Amazon Redshift SQL queries against data that is stored in Amazon S3. With Amazon Redshift Spectrum, you can extend the analytic power of Amazon Redshift beyond the data that is stored on local disks in your data warehouse. You can query vast amounts of data in your Amazon S3 “data lake” without having to go through a tedious and time- consuming extract, transfer, and load (ETL) process. Amazon Redshift Spectrum applies sophisticated query optimization and scales processing across thousands of nodes to deliver fast performance. In this blog post, we have collected 1. Amazon Redshift Spectrum by grouping them into several different functional groups. These guidelines are the product of many interactions and direct project work with Amazon Redshift customers. Amazon Redshift vs. Amazon Athena. AWS customers often ask us: Amazon Athena or Amazon Redshift Spectrum? When should I use one over the other? When to use Amazon Athena. Amazon Athena supports a use case in which you want interactive ad- hoc queries to run against data that is stored in Amazon S3 using SQL. The serverless architecture in Amazon Athena frees you from having to provision a cluster to perform queries. You are charged based on the amount of S3 data scanned by each query. You can get significant cost savings and better performance by compressing, partitioning, or converting your data into a columnar format, which reduces the amount of data that Amazon Athena needs to scan to execute a query. All the major BI tools and SQL clients that use JDBC can be used with Amazon Athena. You can also use Amazon Quick. Sight for easy visualization. When to use Amazon Redshift. We recommend using Amazon Redshift on large sets of structured data. Amazon Redshift Spectrum gives you the freedom to store your data where you want, in the format you want, and have it available for processing when you need it. With Amazon Redshift Spectrum, you don’t have to worry about scaling your cluster. It lets you separate storage and compute, allowing you to scale each independently. You can even run multiple Amazon Redshift clusters against the same Amazon S3 data lake, enabling limitless concurrency. Amazon Redshift Spectrum automatically scales out to thousands of instances. So queries run quickly, whether they are processing a terabyte, a petabyte, or even an exabyte. Set up the test environment. For information about prerequisites and steps to get started in Amazon Redshift Spectrum, see Getting Started with Amazon Redshift Spectrum. You can use any data set to perform the tests to validate the best practices we have outlined in this blog post. One important requirement is that the S3 files for the largest table need to be in three separate data formats: CSV, non- partitioned Parquet as well as partitioned Parquet. How to convert from one file format to another is beyond the scope of this blog post. For more information on how this can be done, check out the following resources: Creating the external schema. Use the Amazon Athena data catalog as the metadata store, and create an external schema named “spectrum” as follows: create external schema spectrum. AWS_ACCOUNT_ID> :role/aod- redshift- role'. The Redshift cluster and the data files in Amazon S3 must be in the same AWS region. Your Redshift cluster needs authorization to access your external data catalog in Amazon Athena and your data files in Amazon S3. You provide that authorization by referencing an AWS Identity and Access Management (IAM) role (e. For more information, see Create an IAM Role for Amazon Redshift. Defining external tables. As examples, an Amazon Redshift Spectrum external table using partitioned Parquet files and another external table using CSV files are defined as follows: CREATE external table spectrum. LINEITEM_PART_PARQ (. L_ORDERKEY BIGINT. L_PARTKEY BIGINT. L_SUPPKEY BIGINT. L_LINENUMBER INT. L_QUANTITY DECIMAL(1. L_EXTENDEDPRICE DECIMAL(1. L_DISCOUNT DECIMAL(1. L_TAX DECIMAL(1. 2,2). L_RETURNFLAG VARCHAR(1. L_LINESTATUS VARCHAR(1. L_COMMITDATE VARCHAR(1. L_RECEIPTDATE VARCHAR(1. L_SHIPINSTRUCT VARCHAR(1. L_SHIPMODE VARCHAR(1. L_COMMENT VARCHAR(1. L_SHIPDATE VARCHAR(1. PARQUET. location 's. CREATE external table spectrum. LINEITEM_CSV (. L_ORDERKEY BIGINT. L_PARTKEY INT. L_SUPPKEY INT. L_LINENUMBER INT. L_QUANTITY DECIMAL(1. L_EXTENDEDPRICE DECIMAL(1. L_DISCOUNT DECIMAL(1. L_TAX DECIMAL(1. 2,2). L_RETURNFLAG VARCHAR(1. L_LINESTATUS VARCHAR(1. L_SHIPDATE VARCHAR(1. L_COMMITDATE VARCHAR(1. L_RECEIPTDATE VARCHAR(1. L_SHIPINSTRUCT VARCHAR(1. L_SHIPMODE VARCHAR(1. L_COMMENT VARCHAR(1. Querying data. To recap, Amazon Redshift Spectrum uses external tables to query data that is stored in Amazon S3. You can query an external table using the same SELECT syntax you use with other Amazon Redshift tables. You can’t write to external tables because they are read- only. You first create an external schema that references an external database, which can reside in either an Amazon Athena data catalog or an Apache Hive metastore, such as Amazon EMR. Then you create an external table in Amazon Redshift using this external schema. You must reference the external table in your SELECT statements by prefixing the table name with the schema name, without needing to create and load the table into Amazon Redshift. The external schema references a database in the external data catalog. This requires an IAM role that authorizes your cluster to access Amazon S3 and Amazon Athena on your behalf. If you would like to perform your tests using Amazon Redshift Spectrum, the following two queries would be a good start: QUERY 1: SELECT l_returnflag. WHERE l_shipdate < = '1. GROUP BY l_returnflag, l_linestatus. ORDER BY l_returnflag, l_linestatus; This query includes only one table and it can be used to highlight the additional processing power provided by the Amazon Redshift Spectrum layer. QUERY 2: SELECT l_orderkey. FROM customer, orders, lineitem. WHERE c_mktsegment = 'BUILDING'. AND c_custkey = o_custkey. AND l_orderkey = o_orderkey. AND o_orderdate < date '1. AND l_shipdate > date '1. GROUP BY l_orderkey, o_orderdate, o_shippriority. ORDER BY revenue desc, o_orderdate. LIMIT 2. 0; This query has joins of three tables and can be very useful to compare Amazon Redshift Spectrum’s performance with that of native Amazon Redshift. Best practices for concurrency. These recommended practices can help you optimize your concurrent workload performance using Amazon Redshift Spectrum. Use Amazon Redshift Spectrum to improve scan- intensive concurrent workloads. Amazon Redshift Spectrum resides on dedicated Amazon Redshift servers that are independent of your cluster. It pushes many compute- intensive tasks, such as predicate filtering and aggregation, down to the Amazon Redshift Spectrum layer, so queries use much less of your cluster’s processing capacity. In addition, Amazon Redshift Spectrum scales intelligently. Based on the demands of your queries, Amazon Redshift Spectrum can potentially use thousands of instances to take advantage of massively parallel processing (MPP). For some use cases of concurrent scan and/or aggregate intensive workloads, Amazon Redshift Spectrum may perform better than native Amazon Redshift on average. The most resource- intensive aspect of any MPP system is the data- load process. This is because it competes with active analytic queries not only for compute resources, but also for locking on the tables through multiversion concurrency control (MVCC). By contrast, if you add new files to an existing external table using Amazon Redshift Spectrum by writing to Amazon S3, and then updating the meta- data to include them as new partitions, you eliminate this workload from the Amazon Redshift cluster. This has an immediate and direct positive impact on concurrency. Use multiple on- demand Amazon Redshift clusters to scale concurrency. Amazon Redshift Spectrum stores data in Amazon S3, which can be accessed by multiple Amazon Redshift clusters to improve concurrent workload performance. A common Amazon Redshift customer scenario is what to do with seasonal spiky, highly concurrent query workloads. Before Amazon Redshift Spectrum, to handle the increased concurrency, customers often had to spin up multiple “read- only” Amazon Redshift clusters by restoring from a snapshot.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. Archives
November 2017
Categories |